SherbyNet - /page/wikibook/entry.mu

Micron Document

<!DOCTYPE html>

<title>Variational Bayesian methods</title>

</head>

Variational Bayesian methods

</h1>

</header>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div role="note" class="hatnote navigation-not-searchable">For the method of approximation in quantum mechanics, see <a href="Variational_method_(quantum_mechanics)" title="Variational method (quantum mechanics)">Variational method (quantum mechanics)</a>.</div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hlist dl,.mw-parser-output .hlist ol,.mw-parser-output .hlist ul{margin:0;padding:0}.mw-parser-output .hlist dd,.mw-parser-output .hlist dt,.mw-parser-output .hlist li{margin:0;display:inline}.mw-parser-output .hlist.inline,.mw-parser-output .hlist.inline dl,.mw-parser-output .hlist.inline ol,.mw-parser-output .hlist.inline ul,.mw-parser-output .hlist dl dl,.mw-parser-output .hlist dl ol,.mw-parser-output .hlist dl ul,.mw-parser-output .hlist ol dl,.mw-parser-output .hlist ol ol,.mw-parser-output .hlist ol ul,.mw-parser-output .hlist ul dl,.mw-parser-output .hlist ul ol,.mw-parser-output .hlist ul ul{display:inline}.mw-parser-output .hlist .mw-empty-li{display:none}.mw-parser-output .hlist dt::after{content:": "}.mw-parser-output .hlist dd::after,.mw-parser-output .hlist li::after{content:" · ";font-weight:bold}.mw-parser-output .hlist dd:last-child::after,.mw-parser-output .hlist dt:last-child::after,.mw-parser-output .hlist li:last-child::after{content:none}.mw-parser-output .hlist dd dd:first-child::before,.mw-parser-output .hlist dd dt:first-child::before,.mw-parser-output .hlist dd li:first-child::before,.mw-parser-output .hlist dt dd:first-child::before,.mw-parser-output .hlist dt dt:first-child::before,.mw-parser-output .hlist dt li:first-child::before,.mw-parser-output .hlist li dd:first-child::before,.mw-parser-output .hlist li dt:first-child::before,.mw-parser-output .hlist li li:first-child::before{content:" (";font-weight:normal}.mw-parser-output .hlist dd dd:last-child::after,.mw-parser-output .hlist dd dt:last-child::after,.mw-parser-output .hlist dd li:last-child::after,.mw-parser-output .hlist dt dd:last-child::after,.mw-parser-output .hlist dt dt:last-child::after,.mw-parser-output .hlist dt li:last-child::after,.mw-parser-output .hlist li dd:last-child::after,.mw-parser-output .hlist li dt:last-child::after,.mw-parser-output .hlist li li:last-child::after{content:")";font-weight:normal}.mw-parser-output .hlist ol{counter-reset:listitem}.mw-parser-output .hlist ol>li{counter-increment:listitem}.mw-parser-output .hlist ol>li::before{content:" "counter(listitem)"\a0 "}.mw-parser-output .hlist dd ol>li:first-child::before,.mw-parser-output .hlist dt ol>li:first-child::before,.mw-parser-output .hlist li ol>li:first-child::before{content:" ("counter(listitem)"\a0 "}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1246091330">

/* start https://en.wikipedia.org/ */

.mw-parser-output .sidebar{width:22em;float:right;clear:right;margin:0.5em 0 1em 1em;background:var(--background-color-neutral-subtle,#f8f9fa);border:1px solid var(--border-color-base,#a2a9b1);padding:0.2em;text-align:center;line-height:1.4em;font-size:88%;border-collapse:collapse;display:table}body.skin-minerva .mw-parser-output .sidebar{display:table!important;float:right!important;margin:0.5em 0 1em 1em!important}.mw-parser-output .sidebar-subgroup{width:100%;margin:0;border-spacing:0}.mw-parser-output .sidebar-left{float:left;clear:left;margin:0.5em 1em 1em 0}.mw-parser-output .sidebar-none{float:none;clear:both;margin:0.5em 1em 1em 0}.mw-parser-output .sidebar-outer-title{padding:0 0.4em 0.2em;font-size:125%;line-height:1.2em;font-weight:bold}.mw-parser-output .sidebar-top-image{padding:0.4em}.mw-parser-output .sidebar-top-caption,.mw-parser-output .sidebar-pretitle-with-top-image,.mw-parser-output .sidebar-caption{padding:0.2em 0.4em 0;line-height:1.2em}.mw-parser-output .sidebar-pretitle{padding:0.4em 0.4em 0;line-height:1.2em}.mw-parser-output .sidebar-title,.mw-parser-output .sidebar-title-with-pretitle{padding:0.2em 0.8em;font-size:145%;line-height:1.2em}.mw-parser-output .sidebar-title-with-pretitle{padding:0.1em 0.4em}.mw-parser-output .sidebar-image{padding:0.2em 0.4em 0.4em}.mw-parser-output .sidebar-heading{padding:0.1em 0.4em}.mw-parser-output .sidebar-content{padding:0 0.5em 0.4em}.mw-parser-output .sidebar-content-with-subgroup{padding:0.1em 0.4em 0.2em}.mw-parser-output .sidebar-above,.mw-parser-output .sidebar-below{padding:0.3em 0.8em;font-weight:bold}.mw-parser-output .sidebar-collapse .sidebar-above,.mw-parser-output .sidebar-collapse .sidebar-below{border-top:1px solid #aaa;border-bottom:1px solid #aaa}.mw-parser-output .sidebar-navbar{text-align:right;font-size:115%;padding:0 0.4em 0.4em}.mw-parser-output .sidebar-list-title{padding:0 0.4em;text-align:left;font-weight:bold;line-height:1.6em;font-size:105%}.mw-parser-output .sidebar-list-title-c{padding:0 0.4em;text-align:center;margin:0 3.3em}@media(max-width:640px){body.mediawiki .mw-parser-output .sidebar{width:100%!important;clear:both;float:none!important;margin-left:0!important;margin-right:0!important}}body.skin--responsive .mw-parser-output .sidebar a>img{max-width:none!important}@media screen{html.skin-theme-clientpref-night .mw-parser-output .sidebar:not(.notheme) .sidebar-list-title,html.skin-theme-clientpref-night .mw-parser-output .sidebar:not(.notheme) .sidebar-title-with-pretitle{background:transparent!important}html.skin-theme-clientpref-night .mw-parser-output .sidebar:not(.notheme) .sidebar-title-with-pretitle a{color:var(--color-progressive)!important}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .sidebar:not(.notheme) .sidebar-list-title,html.skin-theme-clientpref-os .mw-parser-output .sidebar:not(.notheme) .sidebar-title-with-pretitle{background:transparent!important}html.skin-theme-clientpref-os .mw-parser-output .sidebar:not(.notheme) .sidebar-title-with-pretitle a{color:var(--color-progressive)!important}}@media print{body.ns-0 .mw-parser-output .sidebar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><table class="sidebar nomobile nowraplinks hlist"><tbody><tr><td class="sidebar-pretitle">Part of a series on</td></tr><tr><th class="sidebar-title-with-pretitle"><a href="Bayesian_statistics" title="Bayesian statistics">Bayesian statistics</a></th></tr><tr><td class="sidebar-image"></td></tr><tr><td class="sidebar-content">

<a href="Posterior_probability" title="Posterior probability">Posterior</a> = <a href="Likelihood_function" title="Likelihood function">Likelihood</a> × <a href="Prior_probability" title="Prior probability">Prior</a> ÷ <a href="Marginal_likelihood" title="Marginal likelihood">Evidence</a></td>

</tr><tr><th class="sidebar-heading">

Background</th></tr><tr><td class="sidebar-content">

<ul><li><a href="Bayesian_inference" title="Bayesian inference">Bayesian inference</a></li>

<li><a href="Bayesian_probability" title="Bayesian probability">Bayesian probability</a></li>

<li><a href="Bayes'_theorem" title="Bayes' theorem">Bayes' theorem</a></li>

<li><a href="Bernstein%E2%80%93von_Mises_theorem" title="Bernstein–von Mises theorem">Bernstein–von Mises theorem</a></li>

<li><a href="Coherence_(philosophical_gambling_strategy)" class="mw-redirect" title="Coherence (philosophical gambling strategy)">Coherence</a></li>

<li><a href="Cox's_theorem" title="Cox's theorem">Cox's theorem</a></li>

<li><a href="Cromwell's_rule" title="Cromwell's rule">Cromwell's rule</a></li>

<li><a href="Likelihood_principle" title="Likelihood principle">Likelihood principle</a></li>

<li><a href="Principle_of_indifference" title="Principle of indifference">Principle of indifference</a></li>

<li><a href="Principle_of_maximum_entropy" title="Principle of maximum entropy">Principle of maximum entropy</a></li></ul></td>

</tr><tr><th class="sidebar-heading">

Model building</th></tr><tr><td class="sidebar-content">

<ul><li><a href="Conjugate_prior" title="Conjugate prior">Conjugate prior</a></li>

<li><a href="Bayesian_linear_regression" title="Bayesian linear regression">Linear regression</a></li>

<li><a href="Empirical_Bayes_method" title="Empirical Bayes method">Empirical Bayes</a></li>

<li><a href="Bayesian_hierarchical_modeling" title="Bayesian hierarchical modeling">Hierarchical model</a></li></ul></td>

</tr><tr><th class="sidebar-heading">

Posterior approximation</th></tr><tr><td class="sidebar-content">

<ul><li><a href="Markov_chain_Monte_Carlo" title="Markov chain Monte Carlo">Markov chain Monte Carlo</a></li>

<li><a href="Laplace's_approximation" title="Laplace's approximation">Laplace's approximation</a></li>

<li><a href="Integrated_nested_Laplace_approximations" title="Integrated nested Laplace approximations">Integrated nested Laplace approximations</a></li>

<li><a href="Approximate_Bayesian_computation" title="Approximate Bayesian computation">Approximate Bayesian computation</a></li></ul></td>

</tr><tr><th class="sidebar-heading">

Estimators</th></tr><tr><td class="sidebar-content">

<ul><li><a href="Bayesian_estimator" class="mw-redirect" title="Bayesian estimator">Bayesian estimator</a></li>

<li><a href="Credible_interval" title="Credible interval">Credible interval</a></li>

<li><a href="Maximum_a_posteriori_estimation" title="Maximum a posteriori estimation">Maximum a posteriori estimation</a></li></ul></td>

</tr><tr><th class="sidebar-heading">

Evidence approximation</th></tr><tr><td class="sidebar-content">

<ul><li><a href="Evidence_lower_bound" title="Evidence lower bound">Evidence lower bound</a></li>

<li><a href="Nested_sampling_algorithm" title="Nested sampling algorithm">Nested sampling</a></li></ul></td>

</tr><tr><th class="sidebar-heading">

Model evaluation</th></tr><tr><td class="sidebar-content">

<ul><li><a href="Bayes_factor" title="Bayes factor">Bayes factor</a> (<a href="Bayesian_information_criterion" title="Bayesian information criterion">Schwarz criterion</a>)</li>

<li><a href="Bayesian_model_averaging" class="mw-redirect" title="Bayesian model averaging">Model averaging</a></li>

<li><a href="Posterior_predictive_distribution" title="Posterior predictive distribution">Posterior predictive</a></li></ul></td>

</tr><tr><td class="sidebar-below">

<ul><li> <a href="Portal%3AMathematics" title="Portal:Mathematics">Mathematics portal</a></li></ul></td></tr><tr><td class="sidebar-navbar"><style data-mw-deduplicate="TemplateStyles:r1239400231">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbar{display:inline;font-size:88%;font-weight:normal}.mw-parser-output .navbar-collapse{float:left;text-align:left}.mw-parser-output .navbar-boxtext{word-spacing:0}.mw-parser-output .navbar ul{display:inline-block;white-space:nowrap;line-height:inherit}.mw-parser-output .navbar-brackets::before{margin-right:-0.125em;content:"[ "}.mw-parser-output .navbar-brackets::after{margin-left:-0.125em;content:" ]"}.mw-parser-output .navbar li{word-spacing:-0.125em}.mw-parser-output .navbar a>span,.mw-parser-output .navbar a>abbr{text-decoration:inherit}.mw-parser-output .navbar-mini abbr{font-variant:small-caps;border-bottom:none;text-decoration:none;cursor:inherit}.mw-parser-output .navbar-ct-full{font-size:114%;margin:0 7em}.mw-parser-output .navbar-ct-mini{font-size:114%;margin:0 4em}html.skin-theme-clientpref-night .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}@media(prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}}@media print{.mw-parser-output .navbar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style></td></tr></tbody></table>

Variational Bayesian methods are a family of techniques for approximating intractable <a href="Integral" title="Integral">integrals</a> arising in <a href="Bayesian_inference" title="Bayesian inference">Bayesian inference</a> and <a href="Machine_learning" title="Machine learning">machine learning</a>. They are typically used in complex <a href="Statistical_model" title="Statistical model">statistical models</a> consisting of observed variables (usually termed "data") as well as unknown <a href="Parameter" title="Parameter">parameters</a> and <a href="Latent_variable" class="mw-redirect" title="Latent variable">latent variables</a>, with various sorts of relationships among the three types of <a href="Random_variable" title="Random variable">random variables</a>, as might be described by a <a href="Graphical_model" title="Graphical model">graphical model</a>. As typical in Bayesian inference, the parameters and latent variables are grouped together as "unobserved variables". Variational Bayesian methods are primarily used for two purposes:

<ol><li>To provide an analytical approximation to the <a href="Posterior_probability" title="Posterior probability">posterior probability</a> of the unobserved variables, in order to do <a href="Statistical_inference" title="Statistical inference">statistical inference</a> over these variables.</li>

<li>To derive a <a href="Lower_bound" class="mw-redirect" title="Lower bound">lower bound</a> for the <a href="Marginal_likelihood" title="Marginal likelihood">marginal likelihood</a> (sometimes called the evidence) of the observed data (i.e. the <a href="Marginal_probability" class="mw-redirect" title="Marginal probability">marginal probability</a> of the data given the model, with marginalization performed over unobserved variables). This is typically used for performing <a href="Model_selection" title="Model selection">model selection</a>, the general idea being that a higher marginal likelihood for a given model indicates a better fit of the data by that model and hence a greater probability that the model in question was the one that generated the data. (See also the <a href="Bayes_factor" title="Bayes factor">Bayes factor</a> article.)</li></ol>

In the former purpose (that of approximating a posterior probability), variational Bayes is an alternative to <a href="Monte_Carlo_sampling" class="mw-redirect" title="Monte Carlo sampling">Monte Carlo sampling</a> methods—particularly, <a href="Markov_chain_Monte_Carlo" title="Markov chain Monte Carlo">Markov chain Monte Carlo</a> methods such as <a href="Gibbs_sampling" title="Gibbs sampling">Gibbs sampling</a>—for taking a fully Bayesian approach to <a href="Statistical_inference" title="Statistical inference">statistical inference</a> over complex <a href="Probability_distribution" title="Probability distribution">distributions</a> that are difficult to evaluate directly or <a href="Sample_(statistics)" class="mw-redirect" title="Sample (statistics)">sample</a>. In particular, whereas Monte Carlo techniques provide a numerical approximation to the exact posterior using a set of samples, variational Bayes provides a locally-optimal, exact analytical solution to an approximation of the posterior.

Variational Bayes can be seen as an extension of the <a href="Expectation%E2%80%93maximization_algorithm" title="Expectation–maximization algorithm">expectation–maximization</a> (EM) algorithm from <a href="Maximum_likelihood_estimation" title="Maximum likelihood estimation">maximum likelihood</a> (ML) or <a href="Maximum_a_posteriori_estimation" title="Maximum a posteriori estimation">maximum a posteriori</a> (MAP) estimation of the single most probable value of each parameter to fully Bayesian estimation which computes (an approximation to) the entire <a href="Posterior_distribution" class="mw-redirect" title="Posterior distribution">posterior distribution</a> of the parameters and latent variables. As in EM, it finds a set of optimal parameter values, and it has the same alternating structure as does EM, based on a set of interlocked (mutually dependent) equations that cannot be solved analytically.

For many applications, variational Bayes produces solutions of comparable accuracy to Gibbs sampling at greater speed. However, deriving the set of equations used to update the parameters iteratively often requires a large amount of work compared with deriving the comparable Gibbs sampling equations. This is the case even for many models that are conceptually quite simple, as is demonstrated below in the case of a basic non-hierarchical model with only two parameters and no latent variables.

<div class="mw-heading mw-heading2"><h2 id="Mathematical_derivation">Mathematical derivation</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Problem">Problem</h3></div>

In <a href="Calculus_of_variations" title="Calculus of variations">variational</a> inference, the posterior distribution over a set of unobserved variables <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} =\{Z_{1}\dots Z_{n}\}}">

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} =\{Z_{1}\dots Z_{n}\}}</annotation>

</semantics>

</math><img src="./3b5a4e89a042a5d0878b3f590f8ea2d5be8d74a0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:16.002ex; height:2.843ex;" alt="{\displaystyle \mathbf {Z} =\{Z_{1}\dots Z_{n}\}}" loading="lazy"> given some data <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {X} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {X} }</annotation>

</semantics>

</math><img src="./9f75966a2f9d5672136fa9401ee1e75008f95ffd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.019ex; height:2.176ex;" alt="{\displaystyle \mathbf {X} }" loading="lazy"> is approximated by a so-called variational distribution, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q(\mathbf {Z} ):}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q(\mathbf {Z} ):}</annotation>

</semantics>

</math><img src="./c098c21deef6c31cbc33d18797c8ac983121f1bf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:6.574ex; height:2.843ex;" alt="{\displaystyle Q(\mathbf {Z} ):}" loading="lazy">

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )\approx Q(\mathbf {Z} ).}</annotation>

</semantics>

</math><img src="./2c9da53fec9fd72febf9cbd1c6554fb3ba25ebf2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:18.172ex; height:2.843ex;" alt="{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )\approx Q(\mathbf {Z} ).}" loading="lazy"></dd></dl>

The distribution <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q(\mathbf {Z} )}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q(\mathbf {Z} )}</annotation>

</semantics>

</math><img src="./88749053afc1a144f8a7763a9ba2c564cb234f46.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.282ex; height:2.843ex;" alt="{\displaystyle Q(\mathbf {Z} )}" loading="lazy"> is restricted to belong to a family of distributions of simpler form than <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}">

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}</annotation>

</semantics>

</math><img src="./1093996252ce78495fc6eca165bde03055ed469e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.145ex; height:2.843ex;" alt="{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}" loading="lazy"> (e.g. a family of Gaussian distributions), selected with the intention of making <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q(\mathbf {Z} )}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q(\mathbf {Z} )}</annotation>

</semantics>

</math><img src="./88749053afc1a144f8a7763a9ba2c564cb234f46.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.282ex; height:2.843ex;" alt="{\displaystyle Q(\mathbf {Z} )}" loading="lazy"> similar to the true posterior, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}">

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}</annotation>

</semantics>

The similarity (or dissimilarity) is measured in terms of a dissimilarity function <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle d(Q;P)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle d(Q;P)}</annotation>

</semantics>

</math><img src="./37257c5a3c70edaacab240d77b9409e7e7417654.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:7.643ex; height:2.843ex;" alt="{\displaystyle d(Q;P)}" loading="lazy"> and hence inference is performed by selecting the distribution <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q(\mathbf {Z} )}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q(\mathbf {Z} )}</annotation>

</semantics>

</math><img src="./88749053afc1a144f8a7763a9ba2c564cb234f46.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.282ex; height:2.843ex;" alt="{\displaystyle Q(\mathbf {Z} )}" loading="lazy"> that minimizes <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle d(Q;P)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle d(Q;P)}</annotation>

</semantics>

<div class="mw-heading mw-heading3"><h3 id="KL_divergence">KL divergence</h3></div>

The most common type of variational Bayes uses the <a href="Kullback%E2%80%93Leibler_divergence" title="Kullback–Leibler divergence">Kullback–Leibler divergence</a> (KL-divergence) of Q from P as the choice of dissimilarity function. This choice makes this minimization tractable. The KL-divergence is defined as

<msub>

</mrow>

</msub>

</mrow>

</munder>

</mrow>

<mfrac>

<mrow>

</mrow>

</mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle D_{\mathrm {KL} }(Q\parallel P)\triangleq \sum _{\mathbf {Z} }Q(\mathbf {Z} )\log {\frac {Q(\mathbf {Z} )}{P(\mathbf {Z} \mid \mathbf {X} )}}.}</annotation>

</semantics>

</math><img src="./17e7cbe23713c8146b1956a4a953eb60a89425b8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:38.805ex; height:6.843ex;" alt="{\displaystyle D_{\mathrm {KL} }(Q\parallel P)\triangleq \sum _{\mathbf {Z} }Q(\mathbf {Z} )\log {\frac {Q(\mathbf {Z} )}{P(\mathbf {Z} \mid \mathbf {X} )}}.}" loading="lazy"></dd></dl>

Note that Q and P are reversed from what one might expect. This use of reversed KL-divergence is conceptually similar to the <a href="Expectation%E2%80%93maximization_algorithm" title="Expectation–maximization algorithm">expectation–maximization algorithm</a>. (Using the KL-divergence in the other way produces the <a href="Expectation_propagation" title="Expectation propagation">expectation propagation</a> algorithm.)

<div class="mw-heading mw-heading3"><h3 id="Intractability">Intractability</h3></div>

Variational techniques are typically used to form an approximation for:

</mrow>

</mrow>

<mfrac>

<mrow>

</mrow>

</mrow>

</mrow>

</mrow>

<mrow>

</mrow>

</mrow>

</mfrac>

</mrow>

<mfrac>

<mrow>

</mrow>

</mrow>

</mrow>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )={\frac {P(\mathbf {X} \mid \mathbf {Z} )P(\mathbf {Z} )}{P(\mathbf {X} )}}={\frac {P(\mathbf {X} \mid \mathbf {Z} )P(\mathbf {Z} )}{\int _{\mathbf {Z} }P(\mathbf {X} ,\mathbf {Z} ')\,d\mathbf {Z} '}}}</annotation>

</semantics>

</math><img src="./d580617702dc7d0a4e6f8107287f1e481e5478b4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:47.069ex; height:6.843ex;" alt="{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )={\frac {P(\mathbf {X} \mid \mathbf {Z} )P(\mathbf {Z} )}{P(\mathbf {X} )}}={\frac {P(\mathbf {X} \mid \mathbf {Z} )P(\mathbf {Z} )}{\int _{\mathbf {Z} }P(\mathbf {X} ,\mathbf {Z} ')\,d\mathbf {Z} '}}}" loading="lazy"></dd></dl>

The marginalization over <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {X} )}</annotation>

</semantics>

</math><img src="./2f79cd1a2596bb9ce4566be9ebc380b64ede03e4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.574ex; height:2.843ex;" alt="{\displaystyle P(\mathbf {X} )}" loading="lazy"> in the denominator is typically intractable, because, for example, the search space of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> is combinatorially large. Therefore, we seek an approximation, using <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q(\mathbf {Z} )\approx P(\mathbf {Z} \mid \mathbf {X} )}">

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q(\mathbf {Z} )\approx P(\mathbf {Z} \mid \mathbf {X} )}</annotation>

</semantics>

</math><img src="./4cca38629c0343535cc9d79e04e82031a4ddabd3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:17.525ex; height:2.843ex;" alt="{\displaystyle Q(\mathbf {Z} )\approx P(\mathbf {Z} \mid \mathbf {X} )}" loading="lazy">.

<div class="mw-heading mw-heading3"><h3 id="Evidence_lower_bound">Evidence lower bound</h3></div>

<div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Evidence_lower_bound" title="Evidence lower bound">Evidence lower bound</a></div>

Given that <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )={\frac {P(\mathbf {X} ,\mathbf {Z} )}{P(\mathbf {X} )}}}">

</mrow>

</mrow>

<mfrac>

<mrow>

</mrow>

</mrow>

</mrow>

<mrow>

</mrow>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )={\frac {P(\mathbf {X} ,\mathbf {Z} )}{P(\mathbf {X} )}}}</annotation>

</semantics>

</math><img src="./65dc63c3744b999c341fc0635fc325206a9bb63c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.671ex; width:21.322ex; height:6.509ex;" alt="{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )={\frac {P(\mathbf {X} ,\mathbf {Z} )}{P(\mathbf {X} )}}}" loading="lazy">, the KL-divergence above can also be written as

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

</munder>

</mrow>

<mrow>

<mrow>

<mfrac>

<mrow>

</mrow>

</mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</munder>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</munder>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{array}{rl}D_{\mathrm {KL} }(Q\parallel P)&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log {\frac {Q(\mathbf {Z} )}{P(\mathbf {Z} ,\mathbf {X} )}}+\log P(\mathbf {X} )\right]\\&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log Q(\mathbf {Z} )-\log P(\mathbf {Z} ,\mathbf {X} )\right]+\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log P(\mathbf {X} )\right]\end{array}}}</annotation>

</semantics>

</math><img src="./31dbf180c8df24086d1661802b87e61b9ee08efc.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.505ex; width:75.073ex; height:8.176ex;" alt="{\displaystyle {\begin{array}{rl}D_{\mathrm {KL} }(Q\parallel P)&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log {\frac {Q(\mathbf {Z} )}{P(\mathbf {Z} ,\mathbf {X} )}}+\log P(\mathbf {X} )\right]\\&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log Q(\mathbf {Z} )-\log P(\mathbf {Z} ,\mathbf {X} )\right]+\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log P(\mathbf {X} )\right]\end{array}}}" loading="lazy"></dd></dl>

Because <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P(\mathbf {X} )}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {X} )}</annotation>

</semantics>

</math><img src="./2f79cd1a2596bb9ce4566be9ebc380b64ede03e4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.574ex; height:2.843ex;" alt="{\displaystyle P(\mathbf {X} )}" loading="lazy"> is a constant with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</mrow>

</munder>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \sum _{\mathbf {Z} }Q(\mathbf {Z} )=1}</annotation>

</semantics>

</math><img src="./5b82b63d51288faacb531a872414f4c7bd809812.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:13.285ex; height:5.509ex;" alt="{\displaystyle \sum _{\mathbf {Z} }Q(\mathbf {Z} )=1}" loading="lazy"> because <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q(\mathbf {Z} )}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q(\mathbf {Z} )}</annotation>

</semantics>

<msub>

</mrow>

</msub>

</mrow>

</munder>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle D_{\mathrm {KL} }(Q\parallel P)=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log Q(\mathbf {Z} )-\log P(\mathbf {Z} ,\mathbf {X} )\right]+\log P(\mathbf {X} )}</annotation>

</semantics>

</math><img src="./fe7ea8e2d866fa0a91aaab65b66646a27ee2c1ef.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:60.966ex; height:5.509ex;" alt="{\displaystyle D_{\mathrm {KL} }(Q\parallel P)=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log Q(\mathbf {Z} )-\log P(\mathbf {Z} ,\mathbf {X} )\right]+\log P(\mathbf {X} )}" loading="lazy"></dd></dl>

which, according to the definition of <a href="Expected_value" title="Expected value">expected value</a> (for a discrete <a href="Random_variable" title="Random variable">random variable</a>), can be written as follows

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle D_{\mathrm {KL} }(Q\parallel P)=\mathbb {E} _{\mathbf {Q} }\left[\log Q(\mathbf {Z} )-\log P(\mathbf {Z} ,\mathbf {X} )\right]+\log P(\mathbf {X} )}</annotation>

</semantics>

</math><img src="./a8cdf19ada48df35350f6069e8bdb467ea90fa3a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:55.145ex; height:3.009ex;" alt="{\displaystyle D_{\mathrm {KL} }(Q\parallel P)=\mathbb {E} _{\mathbf {Q} }\left[\log Q(\mathbf {Z} )-\log P(\mathbf {Z} ,\mathbf {X} )\right]+\log P(\mathbf {X} )}" loading="lazy"></dd></dl>

which can be rearranged to become

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{array}{rl}\log P(\mathbf {X} )&=D_{\mathrm {KL} }(Q\parallel P)-\mathbb {E} _{\mathbf {Q} }\left[\log Q(\mathbf {Z} )-\log P(\mathbf {Z} ,\mathbf {X} )\right]\\&=D_{\mathrm {KL} }(Q\parallel P)+{\mathcal {L}}(Q)\end{array}}}</annotation>

</semantics>

</math><img src="./a5b75a2614d1b490cab91fd5a0039f5e646d9f74.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.671ex; width:57.574ex; height:6.509ex;" alt="{\displaystyle {\begin{array}{rl}\log P(\mathbf {X} )&=D_{\mathrm {KL} }(Q\parallel P)-\mathbb {E} _{\mathbf {Q} }\left[\log Q(\mathbf {Z} )-\log P(\mathbf {Z} ,\mathbf {X} )\right]\\&=D_{\mathrm {KL} }(Q\parallel P)+{\mathcal {L}}(Q)\end{array}}}" loading="lazy"></dd></dl>

As the log-<a href="Model_evidence" class="mw-redirect" title="Model evidence">evidence</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \log P(\mathbf {X} )}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \log P(\mathbf {X} )}</annotation>

</semantics>

</math><img src="./960216514fcbb365f3720d23b238b30dddf0d12c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:8.933ex; height:2.843ex;" alt="{\displaystyle \log P(\mathbf {X} )}" loading="lazy"> is fixed with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q}</annotation>

</semantics>

</math><img src="./8752c7023b4b3286800fe3238271bbca681219ed.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.838ex; height:2.509ex;" alt="{\displaystyle Q}" loading="lazy">, maximizing the final term <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {L}}(Q)}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {L}}(Q)}</annotation>

</semantics>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q}</annotation>

</semantics>

</math><img src="./8752c7023b4b3286800fe3238271bbca681219ed.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.838ex; height:2.509ex;" alt="{\displaystyle Q}" loading="lazy"> from <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P}</annotation>

</semantics>

</math><img src="./b4dc73bf40314945ff376bd363916a738548d40a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.745ex; height:2.176ex;" alt="{\displaystyle P}" loading="lazy">. By appropriate choice of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q}</annotation>

</semantics>

</math><img src="./8752c7023b4b3286800fe3238271bbca681219ed.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.838ex; height:2.509ex;" alt="{\displaystyle Q}" loading="lazy">, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {L}}(Q)}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {L}}(Q)}</annotation>

</semantics>

</math><img src="./a5b029605b9e0e2a9fdf4f96b0e88944f70bc04b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.251ex; height:2.843ex;" alt="{\displaystyle {\mathcal {L}}(Q)}" loading="lazy"> becomes tractable to compute and to maximize. Hence we have both an analytical approximation <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q}</annotation>

</semantics>

</math><img src="./8752c7023b4b3286800fe3238271bbca681219ed.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.838ex; height:2.509ex;" alt="{\displaystyle Q}" loading="lazy"> for the posterior <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}">

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}</annotation>

</semantics>

</math><img src="./1093996252ce78495fc6eca165bde03055ed469e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.145ex; height:2.843ex;" alt="{\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}" loading="lazy">, and a lower bound <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {L}}(Q)}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {L}}(Q)}</annotation>

</semantics>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \log P(\mathbf {X} )}</annotation>

</semantics>

The lower bound <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {L}}(Q)}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {L}}(Q)}</annotation>

</semantics>

</math><img src="./a5b029605b9e0e2a9fdf4f96b0e88944f70bc04b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.251ex; height:2.843ex;" alt="{\displaystyle {\mathcal {L}}(Q)}" loading="lazy"> is known as the (negative) variational free energy in analogy with <a href="Thermodynamic_free_energy" title="Thermodynamic free energy">thermodynamic free energy</a> because it can also be expressed as a negative energy <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \operatorname {E} _{Q}[\log P(\mathbf {Z} ,\mathbf {X} )]}">

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {E} _{Q}[\log P(\mathbf {Z} ,\mathbf {X} )]}</annotation>

</semantics>

</math><img src="./a7e43c3758a771143c7934709ad3198bf43e1ca6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:16.01ex; height:3.009ex;" alt="{\displaystyle \operatorname {E} _{Q}[\log P(\mathbf {Z} ,\mathbf {X} )]}" loading="lazy"> plus the <a href="Entropy_(information_theory)" title="Entropy (information theory)">entropy</a> of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q}</annotation>

</semantics>

</math><img src="./8752c7023b4b3286800fe3238271bbca681219ed.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.838ex; height:2.509ex;" alt="{\displaystyle Q}" loading="lazy">. The term <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {L}}(Q)}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {L}}(Q)}</annotation>

</semantics>

<div class="mw-heading mw-heading3"><h3 id="Proofs">Proofs</h3></div>

By the generalized <a href="Pythagorean_theorem" title="Pythagorean theorem">Pythagorean theorem</a> of <a href="Bregman_divergence" title="Bregman divergence">Bregman divergence</a>, of which KL-divergence is a special case, it can be shown that:<a href="#cite_note-Tran2018-1">[1]</a><a href="#cite_note-Martin2014-2">[2]</a>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle D_{\mathrm {KL} }(Q\parallel P)\geq D_{\mathrm {KL} }(Q\parallel Q^{*})+D_{\mathrm {KL} }(Q^{*}\parallel P),\forall Q^{*}\in {\mathcal {C}}}</annotation>

</semantics>

</math><img src="./daf718bc8ea60db70698bbcdcae824c89bf382e8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:54.364ex; height:2.843ex;" alt="{\displaystyle D_{\mathrm {KL} }(Q\parallel P)\geq D_{\mathrm {KL} }(Q\parallel Q^{*})+D_{\mathrm {KL} }(Q^{*}\parallel P),\forall Q^{*}\in {\mathcal {C}}}" loading="lazy"></dd></dl>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {C}}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {C}}}</annotation>

</semantics>

<msup>

</mrow>

</msup>

</mrow>

</munder>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q=Q^{*}\triangleq \arg \min _{Q\in {\mathcal {C}}}D_{\mathrm {KL} }(Q\parallel P).}</annotation>

</semantics>

</math><img src="./32799aa37c5fa597e8512093f38925afcf0e95e9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.338ex; width:31.769ex; height:4.509ex;" alt="{\displaystyle Q=Q^{*}\triangleq \arg \min _{Q\in {\mathcal {C}}}D_{\mathrm {KL} }(Q\parallel P).}" loading="lazy"></dd></dl>

In this case, the global minimizer <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q^{*}(\mathbf {Z} )=q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})q^{*}(\mathbf {Z} _{2})=q^{*}(\mathbf {Z} _{2}\mid \mathbf {Z} _{1})q^{*}(\mathbf {Z} _{1}),}">

<msup>

</mrow>

</msup>

</mrow>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q^{*}(\mathbf {Z} )=q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})q^{*}(\mathbf {Z} _{2})=q^{*}(\mathbf {Z} _{2}\mid \mathbf {Z} _{1})q^{*}(\mathbf {Z} _{1}),}</annotation>

</semantics>

</math><img src="./a7b304fd4dffd4220a069db346954ba77dd24572.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:48.955ex; height:2.843ex;" alt="{\displaystyle Q^{*}(\mathbf {Z} )=q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})q^{*}(\mathbf {Z} _{2})=q^{*}(\mathbf {Z} _{2}\mid \mathbf {Z} _{1})q^{*}(\mathbf {Z} _{1}),}" loading="lazy"> with <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} =\{\mathbf {Z_{1}} ,\mathbf {Z_{2}} \},}">

</mrow>

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} =\{\mathbf {Z_{1}} ,\mathbf {Z_{2}} \},}</annotation>

</semantics>

</math><img src="./fc80908d68e13ae65cfdef36b41e6e9f630a1bd7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:14.361ex; height:2.843ex;" alt="{\displaystyle \mathbf {Z} =\{\mathbf {Z_{1}} ,\mathbf {Z_{2}} \},}" loading="lazy"> can be found as follows:<a href="#cite_note-Tran2018-1">[1]</a>

<mtr>

<mtd>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

<mfrac>

<mrow>

</mrow>

</mrow>

<mrow>

</mrow>

</mrow>

</mfrac>

</mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mfrac>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

</mrow>

</mrow>

</mfrac>

</mrow>

<msub>

</mrow>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

<mrow>

<mrow>

<mfrac>

<mrow>

</mrow>

</mrow>

</mrow>

<mrow>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{array}{rl}q^{*}(\mathbf {Z} _{2})&={\frac {P(\mathbf {X} )}{\zeta (\mathbf {X} )}}{\frac {P(\mathbf {Z} _{2}\mid \mathbf {X} )}{\exp(D_{\mathrm {KL} }(q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})\parallel P(\mathbf {Z} _{1}\mid \mathbf {Z} _{2},\mathbf {X} )))}}\\&={\frac {1}{\zeta (\mathbf {X} )}}\exp \mathbb {E} _{q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})}\left(\log {\frac {P(\mathbf {Z} ,\mathbf {X} )}{q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})}}\right),\end{array}}}</annotation>

</semantics>

</math><img src="./3de32be4bdb02133e9a9c3f014de4494459f0ce8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -4.338ex; width:45.588ex; height:9.843ex;" alt="{\displaystyle {\begin{array}{rl}q^{*}(\mathbf {Z} _{2})&={\frac {P(\mathbf {X} )}{\zeta (\mathbf {X} )}}{\frac {P(\mathbf {Z} _{2}\mid \mathbf {X} )}{\exp(D_{\mathrm {KL} }(q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})\parallel P(\mathbf {Z} _{1}\mid \mathbf {Z} _{2},\mathbf {X} )))}}\\&={\frac {1}{\zeta (\mathbf {X} )}}\exp \mathbb {E} _{q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})}\left(\log {\frac {P(\mathbf {Z} ,\mathbf {X} )}{q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})}}\right),\end{array}}}" loading="lazy"></dd></dl>

in which the normalizing constant is:

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mrow>

<msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

<mfrac>

<mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mfrac>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

<msub>

</mrow>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

<mrow>

<mrow>

<mfrac>

<mrow>

</mrow>

</mrow>

</mrow>

<mrow>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{array}{rl}\zeta (\mathbf {X} )&=P(\mathbf {X} )\int _{\mathbf {Z} _{2}}{\frac {P(\mathbf {Z} _{2}\mid \mathbf {X} )}{\exp(D_{\mathrm {KL} }(q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})\parallel P(\mathbf {Z} _{1}\mid \mathbf {Z} _{2},\mathbf {X} )))}}\\&=\int _{\mathbf {Z} _{2}}\exp \mathbb {E} _{q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})}\left(\log {\frac {P(\mathbf {Z} ,\mathbf {X} )}{q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})}}\right).\end{array}}}</annotation>

</semantics>

</math><img src="./f782b665a6fa9baf3f9262553f89b90646d0da69.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -4.338ex; width:46.378ex; height:9.843ex;" alt="{\displaystyle {\begin{array}{rl}\zeta (\mathbf {X} )&=P(\mathbf {X} )\int _{\mathbf {Z} _{2}}{\frac {P(\mathbf {Z} _{2}\mid \mathbf {X} )}{\exp(D_{\mathrm {KL} }(q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})\parallel P(\mathbf {Z} _{1}\mid \mathbf {Z} _{2},\mathbf {X} )))}}\\&=\int _{\mathbf {Z} _{2}}\exp \mathbb {E} _{q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})}\left(\log {\frac {P(\mathbf {Z} ,\mathbf {X} )}{q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})}}\right).\end{array}}}" loading="lazy"></dd></dl>

The term <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \zeta (\mathbf {X} )}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \zeta (\mathbf {X} )}</annotation>

</semantics>

</math><img src="./7e76b933c500ab057152484b5d86dc8e6f6e52a6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.924ex; height:2.843ex;" alt="{\displaystyle \zeta (\mathbf {X} )}" loading="lazy"> is often called the <a href="Model_evidence" class="mw-redirect" title="Model evidence">evidence</a> lower bound (ELBO) in practice, since <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P(\mathbf {X} )\geq \zeta (\mathbf {X} )=\exp({\mathcal {L}}(Q^{*}))}">

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {X} )\geq \zeta (\mathbf {X} )=\exp({\mathcal {L}}(Q^{*}))}</annotation>

</semantics>

</math><img src="./9adb90c98195c0753fe4d2c5663668c34db37b4a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:28.362ex; height:2.843ex;" alt="{\displaystyle P(\mathbf {X} )\geq \zeta (\mathbf {X} )=\exp({\mathcal {L}}(Q^{*}))}" loading="lazy">,<a href="#cite_note-Tran2018-1">[1]</a> as shown above.

By interchanging the roles of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} _{1}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} _{1}}</annotation>

</semantics>

</math><img src="./325b1e03c02002bec004d9f2488f8f52394a9e9c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.688ex; height:2.509ex;" alt="{\displaystyle \mathbf {Z} _{1}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} _{2},}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} _{2},}</annotation>

</semantics>

</math><img src="./55308e109fec1831ab2c6d4cfd000c2aef6ec433.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.335ex; height:2.509ex;" alt="{\displaystyle \mathbf {Z} _{2},}" loading="lazy"> we can iteratively compute the approximated <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q^{*}(\mathbf {Z} _{1})}">

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{*}(\mathbf {Z} _{1})}</annotation>

</semantics>

</math><img src="./0616a50132b72995ea4cd14fbdb9144ccd4e1dfd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:6.631ex; height:2.843ex;" alt="{\displaystyle q^{*}(\mathbf {Z} _{1})}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q^{*}(\mathbf {Z} _{2})}">

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{*}(\mathbf {Z} _{2})}</annotation>

</semantics>

</math><img src="./4febda42b3360fc5705cf395aa2d42092e78f719.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:6.631ex; height:2.843ex;" alt="{\displaystyle q^{*}(\mathbf {Z} _{2})}" loading="lazy"> of the true model's marginals <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P(\mathbf {Z} _{1}\mid \mathbf {X} )}">

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {Z} _{1}\mid \mathbf {X} )}</annotation>

</semantics>

</math><img src="./747dc360a08ecb050150f57992f36750f8f53dbd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.2ex; height:2.843ex;" alt="{\displaystyle P(\mathbf {Z} _{1}\mid \mathbf {X} )}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P(\mathbf {Z} _{2}\mid \mathbf {X} ),}">

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(\mathbf {Z} _{2}\mid \mathbf {X} ),}</annotation>

</semantics>

</math><img src="./7001676a0975da012f8bd01a9db5541e6c004d11.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.846ex; height:2.843ex;" alt="{\displaystyle P(\mathbf {Z} _{2}\mid \mathbf {X} ),}" loading="lazy"> respectively. Although this iterative scheme is guaranteed to converge monotonically,<a href="#cite_note-Tran2018-1">[1]</a> the converged <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q^{*}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q^{*}}</annotation>

</semantics>

</math><img src="./c1785c0a77ab5a06684e8a7ac4f5e59d59ec0319.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.893ex; height:2.676ex;" alt="{\displaystyle Q^{*}}" loading="lazy"> is only a local minimizer of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle D_{\mathrm {KL} }(Q\parallel P)}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle D_{\mathrm {KL} }(Q\parallel P)}</annotation>

</semantics>

</math><img src="./3e7218c8dd2178621cf54e33478b6e3b15da016a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:12.308ex; height:2.843ex;" alt="{\displaystyle D_{\mathrm {KL} }(Q\parallel P)}" loading="lazy">.

If the constrained space <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {C}}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {C}}}</annotation>

</semantics>

</math><img src="./e7b3edab7022ca9e2976651bc59c489513ee9019.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.239ex; height:2.176ex;" alt="{\displaystyle {\mathcal {C}}}" loading="lazy"> is confined within independent space, i.e. <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})=q^{*}(\mathbf {Z_{1}} ),}">

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})=q^{*}(\mathbf {Z_{1}} ),}</annotation>

</semantics>

</math><img src="./e6ddfe24dd50007693f2df773d5424b3934e1a66.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:21.756ex; height:2.843ex;" alt="{\displaystyle q^{*}(\mathbf {Z} _{1}\mid \mathbf {Z} _{2})=q^{*}(\mathbf {Z_{1}} ),}" loading="lazy">the above iterative scheme will become the so-called mean field approximation <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q^{*}(\mathbf {Z} )=q^{*}(\mathbf {Z} _{1})q^{*}(\mathbf {Z} _{2}),}">

<msup>

</mrow>

</msup>

</mrow>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q^{*}(\mathbf {Z} )=q^{*}(\mathbf {Z} _{1})q^{*}(\mathbf {Z} _{2}),}</annotation>

</semantics>

</math><img src="./bdf9163856250789fcce9403d37e8325196b5603.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:23.343ex; height:2.843ex;" alt="{\displaystyle Q^{*}(\mathbf {Z} )=q^{*}(\mathbf {Z} _{1})q^{*}(\mathbf {Z} _{2}),}" loading="lazy">as shown below.

<div class="mw-heading mw-heading2"><h2 id="Mean_field_approximation">Mean field approximation</h2></div>

The variational distribution <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q(\mathbf {Z} )}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q(\mathbf {Z} )}</annotation>

</semantics>

</math><img src="./88749053afc1a144f8a7763a9ba2c564cb234f46.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.282ex; height:2.843ex;" alt="{\displaystyle Q(\mathbf {Z} )}" loading="lazy"> is usually assumed to factorize over some <a href="Partition_of_a_set" title="Partition of a set">partition</a> of the latent variables, i.e. for some partition of the latent variables <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} _{1}\dots \mathbf {Z} _{M}}</annotation>

</semantics>

</math><img src="./a4c797d24283e2ad0a225cd1bf962ac5887b21a9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:9.779ex; height:2.509ex;" alt="{\displaystyle \mathbf {Z} _{1}\dots \mathbf {Z} _{M}}" loading="lazy">,

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q(\mathbf {Z} )=\prod _{i=1}^{M}q_{i}(\mathbf {Z} _{i}\mid \mathbf {X} )}</annotation>

</semantics>

</math><img src="./6094f9818d77c817dcb6e3ccc8168a498a0c6286.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:21.773ex; height:7.343ex;" alt="{\displaystyle Q(\mathbf {Z} )=\prod _{i=1}^{M}q_{i}(\mathbf {Z} _{i}\mid \mathbf {X} )}" loading="lazy"></dd></dl>

It can be shown using the <a href="Calculus_of_variations" title="Calculus of variations">calculus of variations</a> (hence the name "variational Bayes") that the "best" distribution <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q_{j}^{*}}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q_{j}^{*}}</annotation>

</semantics>

</math><img src="./975a794deceb926d031d6ec155b217e4a068ae75.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.338ex; width:2.134ex; height:3.176ex;" alt="{\displaystyle q_{j}^{*}}" loading="lazy"> for each of the factors <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q_{j}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q_{j}}</annotation>

</semantics>

</math><img src="./e0d567ac2d170501680d2efa4c1d71d6a8569ef1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:1.947ex; height:2.343ex;" alt="{\displaystyle q_{j}}" loading="lazy"> (in terms of the distribution minimizing the KL divergence, as described above) satisfies:<a href="#cite_note-Nguyen-3">[3]</a>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

<mfrac>

<msup>

<msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msup>

<mrow>

<msup>

<msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )={\frac {e^{\operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]}}{\int e^{\operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]}\,d\mathbf {Z} _{j}}}}</annotation>

</semantics>

</math><img src="./c433a20d83c8b317b21d3ebe5f6bdc820f564310.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.838ex; width:33.066ex; height:8.509ex;" alt="{\displaystyle q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )={\frac {e^{\operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]}}{\int e^{\operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]}\,d\mathbf {Z} _{j}}}}" loading="lazy"></dd></dl>

<msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]}</annotation>

</semantics>

</math><img src="./8d25b04c50785b1ac3d66bd7e1610b77bc64314d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.505ex; width:15.587ex; height:3.509ex;" alt="{\displaystyle \operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]}" loading="lazy"> is the <a href="Expected_value" title="Expected value">expectation</a> of the logarithm of the <a href="Joint_probability" class="mw-redirect" title="Joint probability">joint probability</a> of the data and latent variables, taken with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q^{*}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{*}}</annotation>

</semantics>

</math><img src="./f8f2b5e5515bb7a24f643d7fcf1550f5eeb04b41.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.134ex; height:2.676ex;" alt="{\displaystyle q^{*}}" loading="lazy"> over all variables not in the partition: refer to Lemma 4.1 of<a href="#cite_note-Yoon2021-4">[4]</a> for a derivation of the distribution <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )}">

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )}</annotation>

</semantics>

In practice, we usually work in terms of logarithms, i.e.:

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</msub>

</mrow>

</mrow>

<mtext>constant</mtext>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ln q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )=\operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]+{\text{constant}}}</annotation>

</semantics>

</math><img src="./87a270648b59bbf964ca39d81d20b7de87d50d25.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.505ex; width:42.963ex; height:3.509ex;" alt="{\displaystyle \ln q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )=\operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]+{\text{constant}}}" loading="lazy"></dd></dl>

The constant in the above expression is related to the <a href="Normalizing_constant" title="Normalizing constant">normalizing constant</a> (the denominator in the expression above for <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q_{j}^{*}}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q_{j}^{*}}</annotation>

</semantics>

</math><img src="./975a794deceb926d031d6ec155b217e4a068ae75.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.338ex; width:2.134ex; height:3.176ex;" alt="{\displaystyle q_{j}^{*}}" loading="lazy">) and is usually reinstated by inspection, as the rest of the expression can usually be recognized as being a known type of distribution (e.g. <a href="Gaussian_distribution" class="mw-redirect" title="Gaussian distribution">Gaussian</a>, <a href="Gamma_distribution" title="Gamma distribution">gamma</a>, etc.).

Using the properties of expectations, the expression <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]}">

<msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]}</annotation>

</semantics>

</math><img src="./8d25b04c50785b1ac3d66bd7e1610b77bc64314d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.505ex; width:15.587ex; height:3.509ex;" alt="{\displaystyle \operatorname {E} _{q_{-j}^{*}}[\ln p(\mathbf {Z} ,\mathbf {X} )]}" loading="lazy"> can usually be simplified into a function of the fixed <a href="Hyperparameter_(Bayesian_statistics)" title="Hyperparameter (Bayesian statistics)">hyperparameters</a> of the <a href="Prior_distribution" class="mw-redirect" title="Prior distribution">prior distributions</a> over the latent variables and of expectations (and sometimes higher <a href="Moment_(mathematics)" title="Moment (mathematics)">moments</a> such as the <a href="Variance" title="Variance">variance</a>) of latent variables not in the current partition (i.e. latent variables not included in <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} _{j}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} _{j}}</annotation>

</semantics>

</math><img src="./5ffbb56db9f29c78324a4758e2dd41dc1c3c0cc1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.544ex; height:2.843ex;" alt="{\displaystyle \mathbf {Z} _{j}}" loading="lazy">). This creates <a href="Circular_dependency" title="Circular dependency">circular dependencies</a> between the parameters of the distributions over variables in one partition and the expectations of variables in the other partitions. This naturally suggests an <a href="Iterative" class="mw-redirect" title="Iterative">iterative</a> algorithm, much like EM (the <a href="Expectation%E2%80%93maximization_algorithm" title="Expectation–maximization algorithm">expectation–maximization algorithm</a>), in which the expectations (and possibly higher moments) of the latent variables are initialized in some fashion (perhaps randomly), and then the parameters of each distribution are computed in turn using the current values of the expectations, after which the expectation of the newly computed distribution is set appropriately according to the computed parameters. An algorithm of this sort is guaranteed to <a href="Limit_of_a_sequence" title="Limit of a sequence">converge</a>.<a href="#cite_note-5">[5]</a>

In other words, for each of the partitions of variables, by simplifying the expression for the distribution over the partition's variables and examining the distribution's <a href="Functional_dependency" title="Functional dependency">functional dependency</a> on the variables in question, the family of the distribution can usually be determined (which in turn determines the value of the constant). The formula for the distribution's parameters will be expressed in terms of the prior distributions' hyperparameters (which are known constants), but also in terms of expectations of functions of variables in other partitions. Usually these expectations can be simplified into functions of expectations of the variables themselves (i.e. the <a href="Mean" title="Mean">means</a>); sometimes expectations of squared variables (which can be related to the <a href="Variance" title="Variance">variance</a> of the variables), or expectations of higher powers (i.e. higher <a href="Moment_(mathematics)" title="Moment (mathematics)">moments</a>) also appear. In most cases, the other variables' distributions will be from known families, and the formulas for the relevant expectations can be looked up. However, those formulas depend on those distributions' parameters, which depend in turn on the expectations about other variables. The result is that the formulas for the parameters of each variable's distributions can be expressed as a series of equations with mutual, <a href="Nonlinear" class="mw-redirect" title="Nonlinear">nonlinear</a> dependencies among the variables. Usually, it is not possible to solve this system of equations directly. However, as described above, the dependencies suggest a simple iterative algorithm, which in most cases is guaranteed to converge. An example will make this process clearer.

<div class="mw-heading mw-heading2"><h2 id="A_duality_formula_for_variational_inference">A duality formula for variational inference</h2></div>

The following theorem is referred to as a duality formula for variational inference.<a href="#cite_note-Yoon2021-4">[4]</a> It explains some important properties of the variational distributions used in variational Bayes methods.

Theorem Consider two <a href="Probability_spaces" class="mw-redirect" title="Probability spaces">probability spaces</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle (\Theta ,{\mathcal {F}},P)}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle (\Theta ,{\mathcal {F}},P)}</annotation>

</semantics>

</math><img src="./a041c954bb66ac75dff73cb36f4451ec412d26ad.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.357ex; height:2.843ex;" alt="{\displaystyle (\Theta ,{\mathcal {F}},P)}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle (\Theta ,{\mathcal {F}},Q)}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle (\Theta ,{\mathcal {F}},Q)}</annotation>

</semantics>

</math><img src="./86c52eea005add082da3daadf597eefd76952f7a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.45ex; height:2.843ex;" alt="{\displaystyle (\Theta ,{\mathcal {F}},Q)}" loading="lazy"> with <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q\ll P}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q\ll P}</annotation>

</semantics>

</math><img src="./dbaff603eec8673703c6850fb36cb4ac6508b717.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:7.198ex; height:2.509ex;" alt="{\displaystyle Q\ll P}" loading="lazy">. Assume that there is a common dominating <a href="Probability_measure" title="Probability measure">probability measure</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \lambda }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \lambda }</annotation>

</semantics>

</math><img src="./b43d0ea3c9c025af1be9128e62a18fa74bedda2a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.355ex; height:2.176ex;" alt="{\displaystyle \lambda }" loading="lazy"> such that <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P\ll \lambda }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P\ll \lambda }</annotation>

</semantics>

</math><img src="./4556d3c282180a8e3a7ad0b1caf97ce75ff9ecda.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:6.715ex; height:2.176ex;" alt="{\displaystyle P\ll \lambda }" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q\ll \lambda }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q\ll \lambda }</annotation>

</semantics>

</math><img src="./4a981d0f9292cace238801c1c3f89d5289a4874f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:6.808ex; height:2.509ex;" alt="{\displaystyle Q\ll \lambda }" loading="lazy">. Let <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle h}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle h}</annotation>

</semantics>

</math><img src="./b26be3e694314bc90c3215047e4a2010c6ee184a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.339ex; height:2.176ex;" alt="{\displaystyle h}" loading="lazy"> denote any real-valued <a href="Random_variable" title="Random variable">random variable</a> on <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle (\Theta ,{\mathcal {F}},P)}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle (\Theta ,{\mathcal {F}},P)}</annotation>

</semantics>

</math><img src="./a041c954bb66ac75dff73cb36f4451ec412d26ad.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.357ex; height:2.843ex;" alt="{\displaystyle (\Theta ,{\mathcal {F}},P)}" loading="lazy"> that satisfies <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle h\in L_{1}(P)}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle h\in L_{1}(P)}</annotation>

</semantics>

</math><img src="./f8525ea64ffd546a49ea8df4605af7adfee57c0f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.371ex; height:2.843ex;" alt="{\displaystyle h\in L_{1}(P)}" loading="lazy">. Then the following equality holds

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \log E_{P}[\exp h]={\text{sup}}_{Q\ll P}\{E_{Q}[h]-D_{\text{KL}}(Q\parallel P)\}.}</annotation>

</semantics>

</math><img src="./6b77d97ef624d72b613687f5ba1dae28f355ea34.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.171ex; width:48.122ex; height:3.176ex;" alt="{\displaystyle \log E_{P}[\exp h]={\text{sup}}_{Q\ll P}\{E_{Q}[h]-D_{\text{KL}}(Q\parallel P)\}.}" loading="lazy"></dd></dl>

Further, the supremum on the right-hand side is attained <a href="If_and_only_if" title="If and only if">if and only if</a> it holds

<mfrac>

<mrow>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

<mfrac>

<mrow>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\frac {q(\theta )}{p(\theta )}}={\frac {\exp h(\theta )}{E_{P}[\exp h]}},}</annotation>

</semantics>

</math><img src="./3aa6d2bde96aa9ae886413e7889fd4496dbe710c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.671ex; width:19.241ex; height:6.509ex;" alt="{\displaystyle {\frac {q(\theta )}{p(\theta )}}={\frac {\exp h(\theta )}{E_{P}[\exp h]}},}" loading="lazy"></dd></dl>

almost surely with respect to probability measure <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q}</annotation>

</semantics>

</math><img src="./8752c7023b4b3286800fe3238271bbca681219ed.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.838ex; height:2.509ex;" alt="{\displaystyle Q}" loading="lazy">, where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p(\theta )=dP/d\lambda }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p(\theta )=dP/d\lambda }</annotation>

</semantics>

</math><img src="./d18cc724b4408221755e1668fe4fb5033790ce34.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.089ex; width:13.952ex; height:2.843ex;" alt="{\displaystyle p(\theta )=dP/d\lambda }" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q(\theta )=dQ/d\lambda }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q(\theta )=dQ/d\lambda }</annotation>

</semantics>

</math><img src="./05dad841d2f927def2e7f4720590e2b3cb9ccd75.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:13.856ex; height:2.843ex;" alt="{\displaystyle q(\theta )=dQ/d\lambda }" loading="lazy"> denote the Radon–Nikodym derivatives of the probability measures <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P}</annotation>

</semantics>

</math><img src="./b4dc73bf40314945ff376bd363916a738548d40a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.745ex; height:2.176ex;" alt="{\displaystyle P}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q}</annotation>

</semantics>

</math><img src="./8752c7023b4b3286800fe3238271bbca681219ed.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.838ex; height:2.509ex;" alt="{\displaystyle Q}" loading="lazy"> with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \lambda }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \lambda }</annotation>

</semantics>

<div class="mw-heading mw-heading2"><h2 id="A_basic_example">A basic example</h2></div>

Consider a simple non-hierarchical Bayesian model consisting of a set of <a href="Independent_identically_distributed" class="mw-redirect" title="Independent identically distributed">i.i.d.</a> observations from a <a href="Gaussian_distribution" class="mw-redirect" title="Gaussian distribution">Gaussian distribution</a>, with unknown <a href="Mean" title="Mean">mean</a> and <a href="Variance" title="Variance">variance</a>.<a href="#cite_note-bishop-6">[6]</a> In the following, we work through this model in great detail to illustrate the workings of the variational Bayes method.

For mathematical convenience, in the following example we work in terms of the <a href="Precision_(statistics)" title="Precision (statistics)">precision</a> — i.e. the reciprocal of the variance (or in a multivariate Gaussian, the inverse of the <a href="Covariance_matrix" title="Covariance matrix">covariance matrix</a>) — rather than the variance itself. (From a theoretical standpoint, precision and variance are equivalent since there is a <a href="One-to-one_correspondence" class="mw-redirect" title="One-to-one correspondence">one-to-one correspondence</a> between the two.)

<div class="mw-heading mw-heading3"><h3 id="The_mathematical_model">The mathematical model</h3></div>

We place <a href="Conjugate_prior" title="Conjugate prior">conjugate prior</a> distributions on the unknown mean <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu }</annotation>

</semantics>

</math><img src="./9fd47b2a39f7a7856952afec1f1db72c67af6161.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:1.402ex; height:2.176ex;" alt="{\displaystyle \mu }" loading="lazy"> and precision <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \tau }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \tau }</annotation>

</semantics>

<mtr>

<mtd>

</mtd>

<mtd>

<mi>Gamma</mi>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

<mtext>number of data points</mtext>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\tau &\sim \operatorname {Gamma} (a_{0},b_{0})\\\mu |\tau &\sim {\mathcal {N}}(\mu _{0},(\lambda _{0}\tau )^{-1})\\\{x_{1},\dots ,x_{N}\}&\sim {\mathcal {N}}(\mu ,\tau ^{-1})\\N&={\text{number of data points}}\end{aligned}}}</annotation>

</semantics>

</math><img src="./56c45767b9084c40aa723534694b01a544a07bdf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -5.671ex; width:38.869ex; height:12.509ex;" alt="{\displaystyle {\begin{aligned}\tau &\sim \operatorname {Gamma} (a_{0},b_{0})\\\mu |\tau &\sim {\mathcal {N}}(\mu _{0},(\lambda _{0}\tau )^{-1})\\\{x_{1},\dots ,x_{N}\}&\sim {\mathcal {N}}(\mu ,\tau ^{-1})\\N&={\text{number of data points}}\end{aligned}}}" loading="lazy"></dd></dl>

The <a href="Hyperparameter_(Bayesian_statistics)" title="Hyperparameter (Bayesian statistics)">hyperparameters</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu _{0},\lambda _{0},a_{0}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu _{0},\lambda _{0},a_{0}}</annotation>

</semantics>

</math><img src="./dd3920a9d8638ba27a0859c5b7830f7d78d41b65.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.217ex; height:2.676ex;" alt="{\displaystyle \mu _{0},\lambda _{0},a_{0}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle b_{0}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle b_{0}}</annotation>

</semantics>

</math><img src="./9e425056f502ca07b103ffbf6ac4720e0f8a01f0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.052ex; height:2.509ex;" alt="{\displaystyle b_{0}}" loading="lazy"> in the prior distributions are fixed, given values. They can be set to small positive numbers to give broad prior distributions indicating ignorance about the prior distributions of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu }</annotation>

</semantics>

</math><img src="./9fd47b2a39f7a7856952afec1f1db72c67af6161.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:1.402ex; height:2.176ex;" alt="{\displaystyle \mu }" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \tau }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \tau }</annotation>

</semantics>

We are given <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle N}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle N}</annotation>

</semantics>

</math><img src="./f5e3890c981ae85503089652feb48b191b57aae3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.064ex; height:2.176ex;" alt="{\displaystyle N}" loading="lazy"> data points <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {X} =\{x_{1},\ldots ,x_{N}\}}">

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {X} =\{x_{1},\ldots ,x_{N}\}}</annotation>

</semantics>

</math><img src="./658770c7e7e8c01fde78df19427112315ff35fba.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:18.026ex; height:2.843ex;" alt="{\displaystyle \mathbf {X} =\{x_{1},\ldots ,x_{N}\}}" loading="lazy"> and our goal is to infer the <a href="Posterior_distribution" class="mw-redirect" title="Posterior distribution">posterior distribution</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q(\mu ,\tau )=p(\mu ,\tau \mid x_{1},\ldots ,x_{N})}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q(\mu ,\tau )=p(\mu ,\tau \mid x_{1},\ldots ,x_{N})}</annotation>

</semantics>

</math><img src="./43d30e0518d9cd59015f1d49ca179c2f3b087234.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:28.752ex; height:2.843ex;" alt="{\displaystyle q(\mu ,\tau )=p(\mu ,\tau \mid x_{1},\ldots ,x_{N})}" loading="lazy"> of the parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu }</annotation>

</semantics>

</math><img src="./9fd47b2a39f7a7856952afec1f1db72c67af6161.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:1.402ex; height:2.176ex;" alt="{\displaystyle \mu }" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \tau .}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \tau .}</annotation>

</semantics>

</math><img src="./871bb01391136d3551c8ea59059e106be2a403cd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.849ex; height:1.676ex;" alt="{\displaystyle \tau .}" loading="lazy">

<div class="mw-heading mw-heading3"><h3 id="The_joint_probability">The joint probability</h3></div>

The <a href="Joint_probability" class="mw-redirect" title="Joint probability">joint probability</a> of all variables can be rewritten as

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p(\mathbf {X} ,\mu ,\tau )=p(\mathbf {X} \mid \mu ,\tau )p(\mu \mid \tau )p(\tau )}</annotation>

</semantics>

</math><img src="./2a477fd72d63519fde4f0369cfe9f171bc385df8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.089ex; width:35.131ex; height:2.843ex;" alt="{\displaystyle p(\mathbf {X} ,\mu ,\tau )=p(\mathbf {X} \mid \mu ,\tau )p(\mu \mid \tau )p(\tau )}" loading="lazy"></dd></dl>

where the individual factors are

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

<mi>Gamma</mi>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}p(\mathbf {X} \mid \mu ,\tau )&=\prod _{n=1}^{N}{\mathcal {N}}(x_{n}\mid \mu ,\tau ^{-1})\\p(\mu \mid \tau )&={\mathcal {N}}\left(\mu \mid \mu _{0},(\lambda _{0}\tau )^{-1}\right)\\p(\tau )&=\operatorname {Gamma} (\tau \mid a_{0},b_{0})\end{aligned}}}</annotation>

</semantics>

</math><img src="./8e8f6901803e5dc60aca07db4faf4dcae3d9fd13.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -6.505ex; width:33.797ex; height:14.176ex;" alt="{\displaystyle {\begin{aligned}p(\mathbf {X} \mid \mu ,\tau )&=\prod _{n=1}^{N}{\mathcal {N}}(x_{n}\mid \mu ,\tau ^{-1})\\p(\mu \mid \tau )&={\mathcal {N}}\left(\mu \mid \mu _{0},(\lambda _{0}\tau )^{-1}\right)\\p(\tau )&=\operatorname {Gamma} (\tau \mid a_{0},b_{0})\end{aligned}}}" loading="lazy"></dd></dl>

where

<mtr>

<mtd>

</mrow>

<msup>

</mrow>

</msup>

</mtd>

<mtd>

<mfrac>

<msqrt>

<msup>

</mrow>

</msup>

</msqrt>

</mfrac>

</mrow>

<msup>

<mfrac>

<mrow>

<msup>

</mrow>

</msup>

</mrow>

<mrow>

<msup>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

<mi>Gamma</mi>

</mtd>

<mtd>

<mfrac>

<mrow>

</mrow>

</mfrac>

</mrow>

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}{\mathcal {N}}(x\mid \mu ,\sigma ^{2})&={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}e^{\frac {-(x-\mu )^{2}}{2\sigma ^{2}}}\\\operatorname {Gamma} (\tau \mid a,b)&={\frac {1}{\Gamma (a)}}b^{a}\tau ^{a-1}e^{-b\tau }\end{aligned}}}</annotation>

</semantics>

</math><img src="./17af5ba66373887df38fdd12e47f4a9edc751d2d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -6.171ex; width:36.118ex; height:13.509ex;" alt="{\displaystyle {\begin{aligned}{\mathcal {N}}(x\mid \mu ,\sigma ^{2})&={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}e^{\frac {-(x-\mu )^{2}}{2\sigma ^{2}}}\\\operatorname {Gamma} (\tau \mid a,b)&={\frac {1}{\Gamma (a)}}b^{a}\tau ^{a-1}e^{-b\tau }\end{aligned}}}" loading="lazy"></dd></dl>

<div class="mw-heading mw-heading3"><h3 id="Factorized_approximation">Factorized approximation</h3></div>

Assume that <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q(\mu ,\tau )=q(\mu )q(\tau )}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q(\mu ,\tau )=q(\mu )q(\tau )}</annotation>

</semantics>

</math><img src="./6c28f5e2b2d3b9847accf422004216bad09d9a9f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:17.976ex; height:2.843ex;" alt="{\displaystyle q(\mu ,\tau )=q(\mu )q(\tau )}" loading="lazy">, i.e. that the posterior distribution factorizes into independent factors for <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu }</annotation>

</semantics>

</math><img src="./9fd47b2a39f7a7856952afec1f1db72c67af6161.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:1.402ex; height:2.176ex;" alt="{\displaystyle \mu }" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \tau }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \tau }</annotation>

</semantics>

</math><img src="./38a7dcde9730ef0853809fefc18d88771f95206c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.202ex; height:1.676ex;" alt="{\displaystyle \tau }" loading="lazy">. This type of assumption underlies the variational Bayesian method. The true posterior distribution does not in fact factor this way (in fact, in this simple case, it is known to be a <a href="Gaussian-gamma_distribution" class="mw-redirect" title="Gaussian-gamma distribution">Gaussian-gamma distribution</a>), and hence the result we obtain will be an approximation.

<div class="mw-heading mw-heading3"><h3 id="Derivation_of_q(μ)">Derivation of q(μ)</h3></div>

Then

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msqrt>

<mfrac>

<mrow>

</mrow>

</mfrac>

</msqrt>

</mrow>

<msup>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

</msup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<msqrt>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

</mrow>

</mfrac>

</msqrt>

</mrow>

<msup>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</msup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<msub>

</mrow>

</msub>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\ln q_{\mu }^{*}(\mu )&=\operatorname {E} _{\tau }\left[\ln p(\mathbf {X} \mid \mu ,\tau )+\ln p(\mu \mid \tau )+\ln p(\tau )\right]+C\\&=\operatorname {E} _{\tau }\left[\ln p(\mathbf {X} \mid \mu ,\tau )\right]+\operatorname {E} _{\tau }\left[\ln p(\mu \mid \tau )\right]+\operatorname {E} _{\tau }\left[\ln p(\tau )\right]+C\\&=\operatorname {E} _{\tau }\left[\ln \prod _{n=1}^{N}{\mathcal {N}}\left(x_{n}\mid \mu ,\tau ^{-1}\right)\right]+\operatorname {E} _{\tau }\left[\ln {\mathcal {N}}\left(\mu \mid \mu _{0},(\lambda _{0}\tau )^{-1}\right)\right]+C_{2}\\&=\operatorname {E} _{\tau }\left[\ln \prod _{n=1}^{N}{\sqrt {\frac {\tau }{2\pi }}}e^{-{\frac {(x_{n}-\mu )^{2}\tau }{2}}}\right]+\operatorname {E} _{\tau }\left[\ln {\sqrt {\frac {\lambda _{0}\tau }{2\pi }}}e^{-{\frac {(\mu -\mu _{0})^{2}\lambda _{0}\tau }{2}}}\right]+C_{2}\\&=\operatorname {E} _{\tau }\left[\sum _{n=1}^{N}\left({\frac {1}{2}}(\ln \tau -\ln 2\pi )-{\frac {(x_{n}-\mu )^{2}\tau }{2}}\right)\right]+\operatorname {E} _{\tau }\left[{\frac {1}{2}}(\ln \lambda _{0}+\ln \tau -\ln 2\pi )-{\frac {(\mu -\mu _{0})^{2}\lambda _{0}\tau }{2}}\right]+C_{2}\\&=\operatorname {E} _{\tau }\left[\sum _{n=1}^{N}-{\frac {(x_{n}-\mu )^{2}\tau }{2}}\right]+\operatorname {E} _{\tau }\left[-{\frac {(\mu -\mu _{0})^{2}\lambda _{0}\tau }{2}}\right]+\operatorname {E} _{\tau }\left[\sum _{n=1}^{N}{\frac {1}{2}}(\ln \tau -\ln 2\pi )\right]+\operatorname {E} _{\tau }\left[{\frac {1}{2}}(\ln \lambda _{0}+\ln \tau -\ln 2\pi )\right]+C_{2}\\&=\operatorname {E} _{\tau }\left[\sum _{n=1}^{N}-{\frac {(x_{n}-\mu )^{2}\tau }{2}}\right]+\operatorname {E} _{\tau }\left[-{\frac {(\mu -\mu _{0})^{2}\lambda _{0}\tau }{2}}\right]+C_{3}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{\sum _{n=1}^{N}(x_{n}-\mu )^{2}+\lambda _{0}(\mu -\mu _{0})^{2}\right\}+C_{3}\end{aligned}}}</annotation>

</semantics>

</math><img src="./c64dbf7b78ed109532c4258d2f03ed35181be164.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -25.505ex; width:121.345ex; height:52.176ex;" alt="{\displaystyle {\begin{aligned}\ln q_{\mu }^{*}(\mu )&=\operatorname {E} _{\tau }\left[\ln p(\mathbf {X} \mid \mu ,\tau )+\ln p(\mu \mid \tau )+\ln p(\tau )\right]+C\\&=\operatorname {E} _{\tau }\left[\ln p(\mathbf {X} \mid \mu ,\tau )\right]+\operatorname {E} _{\tau }\left[\ln p(\mu \mid \tau )\right]+\operatorname {E} _{\tau }\left[\ln p(\tau )\right]+C\\&=\operatorname {E} _{\tau }\left[\ln \prod _{n=1}^{N}{\mathcal {N}}\left(x_{n}\mid \mu ,\tau ^{-1}\right)\right]+\operatorname {E} _{\tau }\left[\ln {\mathcal {N}}\left(\mu \mid \mu _{0},(\lambda _{0}\tau )^{-1}\right)\right]+C_{2}\\&=\operatorname {E} _{\tau }\left[\ln \prod _{n=1}^{N}{\sqrt {\frac {\tau }{2\pi }}}e^{-{\frac {(x_{n}-\mu )^{2}\tau }{2}}}\right]+\operatorname {E} _{\tau }\left[\ln {\sqrt {\frac {\lambda _{0}\tau }{2\pi }}}e^{-{\frac {(\mu -\mu _{0})^{2}\lambda _{0}\tau }{2}}}\right]+C_{2}\\&=\operatorname {E} _{\tau }\left[\sum _{n=1}^{N}\left({\frac {1}{2}}(\ln \tau -\ln 2\pi )-{\frac {(x_{n}-\mu )^{2}\tau }{2}}\right)\right]+\operatorname {E} _{\tau }\left[{\frac {1}{2}}(\ln \lambda _{0}+\ln \tau -\ln 2\pi )-{\frac {(\mu -\mu _{0})^{2}\lambda _{0}\tau }{2}}\right]+C_{2}\\&=\operatorname {E} _{\tau }\left[\sum _{n=1}^{N}-{\frac {(x_{n}-\mu )^{2}\tau }{2}}\right]+\operatorname {E} _{\tau }\left[-{\frac {(\mu -\mu _{0})^{2}\lambda _{0}\tau }{2}}\right]+\operatorname {E} _{\tau }\left[\sum _{n=1}^{N}{\frac {1}{2}}(\ln \tau -\ln 2\pi )\right]+\operatorname {E} _{\tau }\left[{\frac {1}{2}}(\ln \lambda _{0}+\ln \tau -\ln 2\pi )\right]+C_{2}\\&=\operatorname {E} _{\tau }\left[\sum _{n=1}^{N}-{\frac {(x_{n}-\mu )^{2}\tau }{2}}\right]+\operatorname {E} _{\tau }\left[-{\frac {(\mu -\mu _{0})^{2}\lambda _{0}\tau }{2}}\right]+C_{3}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{\sum _{n=1}^{N}(x_{n}-\mu )^{2}+\lambda _{0}(\mu -\mu _{0})^{2}\right\}+C_{3}\end{aligned}}}" loading="lazy"></dd></dl>

In the above derivation, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle C}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle C}</annotation>

</semantics>

</math><img src="./4fc55753007cd3c18576f7933f6f089196732029.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.766ex; height:2.176ex;" alt="{\displaystyle C}" loading="lazy">, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle C_{2}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle C_{2}}</annotation>

</semantics>

</math><img src="./7ec545f7870665e1028b7492746848d149878808.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.716ex; height:2.509ex;" alt="{\displaystyle C_{2}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle C_{3}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle C_{3}}</annotation>

</semantics>

</math><img src="./66e9abeb5057b7afbf88e3169101849354f13c65.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.716ex; height:2.509ex;" alt="{\displaystyle C_{3}}" loading="lazy"> refer to values that are constant with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu }</annotation>

</semantics>

</math><img src="./9fd47b2a39f7a7856952afec1f1db72c67af6161.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:1.402ex; height:2.176ex;" alt="{\displaystyle \mu }" loading="lazy">. Note that the term <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \operatorname {E} _{\tau }[\ln p(\tau )]}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {E} _{\tau }[\ln p(\tau )]}</annotation>

</semantics>

</math><img src="./ccf212e8b1c0bdaf7c00735ff771c5f117572089.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.466ex; height:2.843ex;" alt="{\displaystyle \operatorname {E} _{\tau }[\ln p(\tau )]}" loading="lazy"> is not a function of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu }</annotation>

</semantics>

</math><img src="./9fd47b2a39f7a7856952afec1f1db72c67af6161.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:1.402ex; height:2.176ex;" alt="{\displaystyle \mu }" loading="lazy"> and will have the same value regardless of the value of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu }</annotation>

</semantics>

</math><img src="./9fd47b2a39f7a7856952afec1f1db72c67af6161.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:1.402ex; height:2.176ex;" alt="{\displaystyle \mu }" loading="lazy">. Hence in line 3 we can absorb it into the <a href="Constant_term" title="Constant term">constant term</a> at the end. We do the same thing in line 7.

The last line is simply a quadratic polynomial in <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu }</annotation>

</semantics>

</math><img src="./9fd47b2a39f7a7856952afec1f1db72c67af6161.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:1.402ex; height:2.176ex;" alt="{\displaystyle \mu }" loading="lazy">. Since this is the logarithm of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q_{\mu }^{*}(\mu )}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q_{\mu }^{*}(\mu )}</annotation>

</semantics>

</math><img src="./19d118ecdb4a6016ad4c038245ffdd70634af87a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:5.471ex; height:3.009ex;" alt="{\displaystyle q_{\mu }^{*}(\mu )}" loading="lazy">, we can see that <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q_{\mu }^{*}(\mu )}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q_{\mu }^{*}(\mu )}</annotation>

</semantics>

</math><img src="./19d118ecdb4a6016ad4c038245ffdd70634af87a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:5.471ex; height:3.009ex;" alt="{\displaystyle q_{\mu }^{*}(\mu )}" loading="lazy"> itself is a <a href="Gaussian_distribution" class="mw-redirect" title="Gaussian distribution">Gaussian distribution</a>.

With a certain amount of tedious math (expanding the squares inside of the braces, separating out and grouping the terms involving <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu }</annotation>

</semantics>

</math><img src="./9fd47b2a39f7a7856952afec1f1db72c67af6161.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:1.402ex; height:2.176ex;" alt="{\displaystyle \mu }" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu ^{2}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu ^{2}}</annotation>

</semantics>

</math><img src="./155c5b5630673f7c1e819577a3ddf1f45cd6cfc1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:2.456ex; height:3.176ex;" alt="{\displaystyle \mu ^{2}}" loading="lazy"> and <a href="Completing_the_square" title="Completing the square">completing the square</a> over <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu }</annotation>

</semantics>

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<msup>

</mrow>

</msup>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<msup>

</mrow>

</msup>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

<msup>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

</msup>

<msup>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<msup>

</mrow>

</msup>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

<msup>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msup>

<mrow>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

</mfrac>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

<mrow>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\ln q_{\mu }^{*}(\mu )&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{\sum _{n=1}^{N}(x_{n}-\mu )^{2}+\lambda _{0}(\mu -\mu _{0})^{2}\right\}+C_{3}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{\sum _{n=1}^{N}(x_{n}^{2}-2x_{n}\mu +\mu ^{2})+\lambda _{0}(\mu ^{2}-2\mu _{0}\mu +\mu _{0}^{2})\right\}+C_{3}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{\left(\sum _{n=1}^{N}x_{n}^{2}\right)-2\left(\sum _{n=1}^{N}x_{n}\right)\mu +\left(\sum _{n=1}^{N}\mu ^{2}\right)+\lambda _{0}\mu ^{2}-2\lambda _{0}\mu _{0}\mu +\lambda _{0}\mu _{0}^{2}\right\}+C_{3}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\mu ^{2}-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\mu +\left(\sum _{n=1}^{N}x_{n}^{2}\right)+\lambda _{0}\mu _{0}^{2}\right\}+C_{3}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\mu ^{2}-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\mu \right\}+C_{4}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\mu ^{2}-2\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)(\lambda _{0}+N)\mu \right\}+C_{4}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\left(\mu ^{2}-2\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)\mu \right)\right\}+C_{4}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\left(\mu ^{2}-2\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)\mu +\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)^{2}-\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)^{2}\right)\right\}+C_{4}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\left(\mu ^{2}-2\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)\mu +\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)^{2}\right)\right\}+C_{5}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\left(\mu -{\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)^{2}\right\}+C_{5}\\&=-{\frac {1}{2}}(\lambda _{0}+N)\operatorname {E} _{\tau }[\tau ]\left(\mu -{\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)^{2}+C_{5}\end{aligned}}}</annotation>

</semantics>

</math><img src="./b95121a849ebbf25ba5413565ef4696267ba0e42.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -43.124ex; margin-bottom: -0.214ex; width:122.323ex; height:87.843ex;" alt="{\displaystyle {\begin{aligned}\ln q_{\mu }^{*}(\mu )&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{\sum _{n=1}^{N}(x_{n}-\mu )^{2}+\lambda _{0}(\mu -\mu _{0})^{2}\right\}+C_{3}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{\sum _{n=1}^{N}(x_{n}^{2}-2x_{n}\mu +\mu ^{2})+\lambda _{0}(\mu ^{2}-2\mu _{0}\mu +\mu _{0}^{2})\right\}+C_{3}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{\left(\sum _{n=1}^{N}x_{n}^{2}\right)-2\left(\sum _{n=1}^{N}x_{n}\right)\mu +\left(\sum _{n=1}^{N}\mu ^{2}\right)+\lambda _{0}\mu ^{2}-2\lambda _{0}\mu _{0}\mu +\lambda _{0}\mu _{0}^{2}\right\}+C_{3}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\mu ^{2}-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\mu +\left(\sum _{n=1}^{N}x_{n}^{2}\right)+\lambda _{0}\mu _{0}^{2}\right\}+C_{3}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\mu ^{2}-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\mu \right\}+C_{4}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\mu ^{2}-2\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)(\lambda _{0}+N)\mu \right\}+C_{4}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\left(\mu ^{2}-2\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)\mu \right)\right\}+C_{4}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\left(\mu ^{2}-2\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)\mu +\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)^{2}-\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)^{2}\right)\right\}+C_{4}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\left(\mu ^{2}-2\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)\mu +\left({\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)^{2}\right)\right\}+C_{5}\\&=-{\frac {\operatorname {E} _{\tau }[\tau ]}{2}}\left\{(\lambda _{0}+N)\left(\mu -{\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)^{2}\right\}+C_{5}\\&=-{\frac {1}{2}}(\lambda _{0}+N)\operatorname {E} _{\tau }[\tau ]\left(\mu -{\frac {\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}}{\lambda _{0}+N}}\right)^{2}+C_{5}\end{aligned}}}" loading="lazy"></dd></dl>

Note that all of the above steps can be shortened by using the formula for the <a href="Normal_distribution#Sum_of_two_quadratics" title="Normal distribution">sum of two quadratics</a>.

In other words:

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<mover>

</mover>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mover>

</mover>

</mrow>

</mtd>

<mtd>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}q_{\mu }^{*}(\mu )&\sim {\mathcal {N}}(\mu \mid \mu _{N},\lambda _{N}^{-1})\\\mu _{N}&={\frac {\lambda _{0}\mu _{0}+N{\bar {x}}}{\lambda _{0}+N}}\\\lambda _{N}&=(\lambda _{0}+N)\operatorname {E} _{\tau }[\tau ]\\{\bar {x}}&={\frac {1}{N}}\sum _{n=1}^{N}x_{n}\end{aligned}}}</annotation>

</semantics>

</math><img src="./2d299b83cbb420dff824c2914458942a0fb39859.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -9.163ex; margin-bottom: -0.175ex; width:24.56ex; height:19.843ex;" alt="{\displaystyle {\begin{aligned}q_{\mu }^{*}(\mu )&\sim {\mathcal {N}}(\mu \mid \mu _{N},\lambda _{N}^{-1})\\\mu _{N}&={\frac {\lambda _{0}\mu _{0}+N{\bar {x}}}{\lambda _{0}+N}}\\\lambda _{N}&=(\lambda _{0}+N)\operatorname {E} _{\tau }[\tau ]\\{\bar {x}}&={\frac {1}{N}}\sum _{n=1}^{N}x_{n}\end{aligned}}}" loading="lazy"></dd></dl>

<div class="mw-heading mw-heading3"><h3 id="Derivation_of_q(τ)">Derivation of q(τ)</h3></div>

The derivation of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q_{\tau }^{*}(\tau )}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q_{\tau }^{*}(\tau )}</annotation>

</semantics>

</math><img src="./1e1e42dc79dac09f18a284556d7478a008c28503.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.145ex; height:2.843ex;" alt="{\displaystyle q_{\tau }^{*}(\tau )}" loading="lazy"> is similar to above, although we omit some of the details for the sake of brevity.

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

<mtext>constant</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

<mfrac>

</mfrac>

</mrow>

<mfrac>

</mfrac>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

<mtext>constant</mtext>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\ln q_{\tau }^{*}(\tau )&=\operatorname {E} _{\mu }[\ln p(\mathbf {X} \mid \mu ,\tau )+\ln p(\mu \mid \tau )]+\ln p(\tau )+{\text{constant}}\\&=(a_{0}-1)\ln \tau -b_{0}\tau +{\frac {1}{2}}\ln \tau +{\frac {N}{2}}\ln \tau -{\frac {\tau }{2}}\operatorname {E} _{\mu }\left[\sum _{n=1}^{N}(x_{n}-\mu )^{2}+\lambda _{0}(\mu -\mu _{0})^{2}\right]+{\text{constant}}\end{aligned}}}</annotation>

</semantics>

</math><img src="./58986c8046658988c363da1009d0beedc83ce01c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -4.838ex; width:97.948ex; height:10.843ex;" alt="{\displaystyle {\begin{aligned}\ln q_{\tau }^{*}(\tau )&=\operatorname {E} _{\mu }[\ln p(\mathbf {X} \mid \mu ,\tau )+\ln p(\mu \mid \tau )]+\ln p(\tau )+{\text{constant}}\\&=(a_{0}-1)\ln \tau -b_{0}\tau +{\frac {1}{2}}\ln \tau +{\frac {N}{2}}\ln \tau -{\frac {\tau }{2}}\operatorname {E} _{\mu }\left[\sum _{n=1}^{N}(x_{n}-\mu )^{2}+\lambda _{0}(\mu -\mu _{0})^{2}\right]+{\text{constant}}\end{aligned}}}" loading="lazy"></dd></dl>

Exponentiating both sides, we can see that <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q_{\tau }^{*}(\tau )}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q_{\tau }^{*}(\tau )}</annotation>

</semantics>

</math><img src="./1e1e42dc79dac09f18a284556d7478a008c28503.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.145ex; height:2.843ex;" alt="{\displaystyle q_{\tau }^{*}(\tau )}" loading="lazy"> is a <a href="Gamma_distribution" title="Gamma distribution">gamma distribution</a>. Specifically:

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

<mi>Gamma</mi>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

<mrow>

</mrow>

</mfrac>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}q_{\tau }^{*}(\tau )&\sim \operatorname {Gamma} (\tau \mid a_{N},b_{N})\\a_{N}&=a_{0}+{\frac {N+1}{2}}\\b_{N}&=b_{0}+{\frac {1}{2}}\operatorname {E} _{\mu }\left[\sum _{n=1}^{N}(x_{n}-\mu )^{2}+\lambda _{0}(\mu -\mu _{0})^{2}\right]\end{aligned}}}</annotation>

</semantics>

</math><img src="./f6ca127beab48a5d09ca7e57f8826ed863d01727.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -7.2ex; margin-bottom: -0.305ex; width:49.61ex; height:16.176ex;" alt="{\displaystyle {\begin{aligned}q_{\tau }^{*}(\tau )&\sim \operatorname {Gamma} (\tau \mid a_{N},b_{N})\\a_{N}&=a_{0}+{\frac {N+1}{2}}\\b_{N}&=b_{0}+{\frac {1}{2}}\operatorname {E} _{\mu }\left[\sum _{n=1}^{N}(x_{n}-\mu )^{2}+\lambda _{0}(\mu -\mu _{0})^{2}\right]\end{aligned}}}" loading="lazy"></dd></dl>

<div class="mw-heading mw-heading3"><h3 id="Algorithm_for_computing_the_parameters">Algorithm for computing the parameters</h3></div>

Let us recap the conclusions from the previous sections:

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<mover>

</mover>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mover>

</mover>

</mrow>

</mtd>

<mtd>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

</semantics>

and

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

<mi>Gamma</mi>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

<mrow>

</mrow>

</mfrac>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

</semantics>

In each case, the parameters for the distribution over one of the variables depend on expectations taken with respect to the other variable. We can expand the expectations, using the standard formulas for the expectations of moments of the Gaussian and gamma distributions:

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<mfrac>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mfrac>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<mrow>

<msup>

</mrow>

</msup>

</mrow>

</mtd>

<mtd>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

<mrow>

<mrow>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</mtd>

<mtd>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\operatorname {E} [\tau \mid a_{N},b_{N}]&={\frac {a_{N}}{b_{N}}}\\\operatorname {E} \left[\mu \mid \mu _{N},\lambda _{N}^{-1}\right]&=\mu _{N}\\\operatorname {E} \left[X^{2}\right]&=\operatorname {Var} (X)+(\operatorname {E} [X])^{2}\\\operatorname {E} \left[\mu ^{2}\mid \mu _{N},\lambda _{N}^{-1}\right]&=\lambda _{N}^{-1}+\mu _{N}^{2}\end{aligned}}}</annotation>

</semantics>

</math><img src="./917241f915e88f88f8e3ff74096507c6efe8551c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -7.338ex; width:37.747ex; height:15.843ex;" alt="{\displaystyle {\begin{aligned}\operatorname {E} [\tau \mid a_{N},b_{N}]&={\frac {a_{N}}{b_{N}}}\\\operatorname {E} \left[\mu \mid \mu _{N},\lambda _{N}^{-1}\right]&=\mu _{N}\\\operatorname {E} \left[X^{2}\right]&=\operatorname {Var} (X)+(\operatorname {E} [X])^{2}\\\operatorname {E} \left[\mu ^{2}\mid \mu _{N},\lambda _{N}^{-1}\right]&=\lambda _{N}^{-1}+\mu _{N}^{2}\end{aligned}}}" loading="lazy"></dd></dl>

Applying these formulas to the above equations is trivial in most cases, but the equation for <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle b_{N}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle b_{N}}</annotation>

</semantics>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}b_{N}&=b_{0}+{\frac {1}{2}}\operatorname {E} _{\mu }\left[\sum _{n=1}^{N}(x_{n}-\mu )^{2}+\lambda _{0}(\mu -\mu _{0})^{2}\right]\\&=b_{0}+{\frac {1}{2}}\operatorname {E} _{\mu }\left[(\lambda _{0}+N)\mu ^{2}-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\mu +\left(\sum _{n=1}^{N}x_{n}^{2}\right)+\lambda _{0}\mu _{0}^{2}\right]\\&=b_{0}+{\frac {1}{2}}\left[(\lambda _{0}+N)\operatorname {E} _{\mu }[\mu ^{2}]-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\operatorname {E} _{\mu }[\mu ]+\left(\sum _{n=1}^{N}x_{n}^{2}\right)+\lambda _{0}\mu _{0}^{2}\right]\\&=b_{0}+{\frac {1}{2}}\left[(\lambda _{0}+N)\left(\lambda _{N}^{-1}+\mu _{N}^{2}\right)-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\mu _{N}+\left(\sum _{n=1}^{N}x_{n}^{2}\right)+\lambda _{0}\mu _{0}^{2}\right]\\\end{aligned}}}</annotation>

</semantics>

</math><img src="./16171a58ee7ad1a9dca35de3637f8034197b22a6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -14.671ex; width:83.854ex; height:30.509ex;" alt="{\displaystyle {\begin{aligned}b_{N}&=b_{0}+{\frac {1}{2}}\operatorname {E} _{\mu }\left[\sum _{n=1}^{N}(x_{n}-\mu )^{2}+\lambda _{0}(\mu -\mu _{0})^{2}\right]\\&=b_{0}+{\frac {1}{2}}\operatorname {E} _{\mu }\left[(\lambda _{0}+N)\mu ^{2}-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\mu +\left(\sum _{n=1}^{N}x_{n}^{2}\right)+\lambda _{0}\mu _{0}^{2}\right]\\&=b_{0}+{\frac {1}{2}}\left[(\lambda _{0}+N)\operatorname {E} _{\mu }[\mu ^{2}]-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\operatorname {E} _{\mu }[\mu ]+\left(\sum _{n=1}^{N}x_{n}^{2}\right)+\lambda _{0}\mu _{0}^{2}\right]\\&=b_{0}+{\frac {1}{2}}\left[(\lambda _{0}+N)\left(\lambda _{N}^{-1}+\mu _{N}^{2}\right)-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\mu _{N}+\left(\sum _{n=1}^{N}x_{n}^{2}\right)+\lambda _{0}\mu _{0}^{2}\right]\\\end{aligned}}}" loading="lazy"></dd></dl>

We can then write the parameter equations as follows, without any expectations:

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<mover>

</mover>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mfrac>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<mover>

</mover>

</mrow>

</mtd>

<mtd>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

<mrow>

</mrow>

</mfrac>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\mu _{N}&={\frac {\lambda _{0}\mu _{0}+N{\bar {x}}}{\lambda _{0}+N}}\\\lambda _{N}&=(\lambda _{0}+N){\frac {a_{N}}{b_{N}}}\\{\bar {x}}&={\frac {1}{N}}\sum _{n=1}^{N}x_{n}\\a_{N}&=a_{0}+{\frac {N+1}{2}}\\b_{N}&=b_{0}+{\frac {1}{2}}\left[(\lambda _{0}+N)\left(\lambda _{N}^{-1}+\mu _{N}^{2}\right)-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\mu _{N}+\left(\sum _{n=1}^{N}x_{n}^{2}\right)+\lambda _{0}\mu _{0}^{2}\right]\end{aligned}}}</annotation>

</semantics>

</math><img src="./fe12241932dc8dfb6a202e043b02cd60b3f593a6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -15.338ex; width:84.258ex; height:31.843ex;" alt="{\displaystyle {\begin{aligned}\mu _{N}&={\frac {\lambda _{0}\mu _{0}+N{\bar {x}}}{\lambda _{0}+N}}\\\lambda _{N}&=(\lambda _{0}+N){\frac {a_{N}}{b_{N}}}\\{\bar {x}}&={\frac {1}{N}}\sum _{n=1}^{N}x_{n}\\a_{N}&=a_{0}+{\frac {N+1}{2}}\\b_{N}&=b_{0}+{\frac {1}{2}}\left[(\lambda _{0}+N)\left(\lambda _{N}^{-1}+\mu _{N}^{2}\right)-2\left(\lambda _{0}\mu _{0}+\sum _{n=1}^{N}x_{n}\right)\mu _{N}+\left(\sum _{n=1}^{N}x_{n}^{2}\right)+\lambda _{0}\mu _{0}^{2}\right]\end{aligned}}}" loading="lazy"></dd></dl>

Note that there are circular dependencies among the formulas for <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \lambda _{N}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \lambda _{N}}</annotation>

</semantics>

</math><img src="./595111953b67b8b3128ad3b7e568ff3f0f997be4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.047ex; height:2.509ex;" alt="{\displaystyle \lambda _{N}}" loading="lazy">and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle b_{N}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle b_{N}}</annotation>

</semantics>

</math><img src="./bf4e4e22fd2dea61c6ff23552caf55c7c772368b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.689ex; height:2.509ex;" alt="{\displaystyle b_{N}}" loading="lazy">. This naturally suggests an <a href="Expectation%E2%80%93maximization_algorithm" title="Expectation–maximization algorithm">EM</a>-like algorithm:

<ol><li>Compute <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \sum _{n=1}^{N}x_{n}}">

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \sum _{n=1}^{N}x_{n}}</annotation>

</semantics>

</math><img src="./7b1ecbd29b6f06d832ee19bff142d17e822921a0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:6.29ex; height:7.343ex;" alt="{\displaystyle \sum _{n=1}^{N}x_{n}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \sum _{n=1}^{N}x_{n}^{2}.}">

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \sum _{n=1}^{N}x_{n}^{2}.}</annotation>

</semantics>

</math><img src="./136329355406e35d28ec3cd668d78e35a7427381.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:6.937ex; height:7.343ex;" alt="{\displaystyle \sum _{n=1}^{N}x_{n}^{2}.}" loading="lazy"> Use these values to compute <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mu _{N}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu _{N}}</annotation>

</semantics>

</math><img src="./c2136e1026c8a9df2da1fdbe61a6e7d183b387e9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:3.093ex; height:2.176ex;" alt="{\displaystyle \mu _{N}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle a_{N}.}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle a_{N}.}</annotation>

</semantics>

</math><img src="./35f87ecc4888664d11d4684e4a90f05383d6e06e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.568ex; height:2.009ex;" alt="{\displaystyle a_{N}.}" loading="lazy"></li>

<li>Initialize <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \lambda _{N}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \lambda _{N}}</annotation>

</semantics>

<li>Use the current value of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \lambda _{N},}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \lambda _{N},}</annotation>

</semantics>

</math><img src="./8cd7b32f2dee88fe802c33abb0feae5dd2811a32.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.694ex; height:2.509ex;" alt="{\displaystyle \lambda _{N},}" loading="lazy"> along with the known values of the other parameters, to compute <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle b_{N}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle b_{N}}</annotation>

</semantics>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle b_{N},}</annotation>

</semantics>

</math><img src="./c4096c740b6bf2d164be0e5b5ed361c37ab1f434.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.336ex; height:2.509ex;" alt="{\displaystyle b_{N},}" loading="lazy"> along with the known values of the other parameters, to compute <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \lambda _{N}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \lambda _{N}}</annotation>

</semantics>

<li>Repeat the last two steps until convergence (i.e. until neither value has changed more than some small amount).</li></ol>

We then have values for the hyperparameters of the approximating distributions of the posterior parameters, which we can use to compute any properties we want of the posterior — e.g. its mean and variance, a 95% highest-density region (the smallest interval that includes 95% of the total probability), etc.

It can be shown that this algorithm is guaranteed to converge to a local maximum.

Note also that the posterior distributions have the same form as the corresponding prior distributions. We did not assume this; the only assumption we made was that the distributions factorize, and the form of the distributions followed naturally. It turns out (see below) that the fact that the posterior distributions have the same form as the prior distributions is not a coincidence, but a general result whenever the prior distributions are members of the <a href="Exponential_family" title="Exponential family">exponential family</a>, which is the case for most of the standard distributions.

<div class="mw-heading mw-heading2"><h2 id="Further_discussion">Further discussion</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Step-by-step_recipe">Step-by-step recipe</h3></div>

The above example shows the method by which the variational-Bayesian approximation to a <a href="Posterior_probability" title="Posterior probability">posterior probability</a> density in a given <a href="Bayesian_network" title="Bayesian network">Bayesian network</a> is derived:

<ol><li>Describe the network with a <a href="Graphical_model" title="Graphical model">graphical model</a>, identifying the observed variables (data) <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {X} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {X} }</annotation>

</semantics>

</math><img src="./9f75966a2f9d5672136fa9401ee1e75008f95ffd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.019ex; height:2.176ex;" alt="{\displaystyle \mathbf {X} }" loading="lazy"> and unobserved variables (<a href="Parameter" title="Parameter">parameters</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\Theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\Theta }}}</annotation>

</semantics>

</math><img src="./f2749cbb9bd4c2ce7c4787c264216fedfff3ee81.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.078ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\Theta }}}" loading="lazy"> and <a href="Latent_variable" class="mw-redirect" title="Latent variable">latent variables</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy">) and their <a href="Conditional_probability_distribution" title="Conditional probability distribution">conditional probability distributions</a>. Variational Bayes will then construct an approximation to the posterior probability <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p(\mathbf {Z} ,{\boldsymbol {\Theta }}\mid \mathbf {X} )}">

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p(\mathbf {Z} ,{\boldsymbol {\Theta }}\mid \mathbf {X} )}</annotation>

</semantics>

</math><img src="./2a49582d5ce49e43550e0fe466ad7e6214b8ef92.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.089ex; width:11.77ex; height:2.843ex;" alt="{\displaystyle p(\mathbf {Z} ,{\boldsymbol {\Theta }}\mid \mathbf {X} )}" loading="lazy">. The approximation has the basic property that it is a factorized distribution, i.e. a product of two or more <a href="Statistical_independence" class="mw-redirect" title="Statistical independence">independent</a> distributions over disjoint subsets of the unobserved variables.</li>

<li>Partition the unobserved variables into two or more subsets, over which the independent factors will be derived. There is no universal procedure for doing this; creating too many subsets yields a poor approximation, while creating too few makes the entire variational Bayes procedure intractable. Typically, the first split is to separate the parameters and latent variables; often, this is enough by itself to produce a tractable result. Assume that the partitions are called <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} _{1},\ldots ,\mathbf {Z} _{M}}">

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} _{1},\ldots ,\mathbf {Z} _{M}}</annotation>

</semantics>

</math><img src="./cca2414b6018df7d6ac0f10c402bc02f11cdb10b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:11.459ex; height:2.509ex;" alt="{\displaystyle \mathbf {Z} _{1},\ldots ,\mathbf {Z} _{M}}" loading="lazy">.</li>

<li>For a given partition <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} _{j}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} _{j}}</annotation>

</semantics>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )}</annotation>

</semantics>

</math><img src="./27f85d66607ecb5faa10f38986cb03faf9bc5096.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.338ex; width:10.443ex; height:3.343ex;" alt="{\displaystyle q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )}" loading="lazy"> using the basic equation <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ln q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )=\operatorname {E} _{i\neq j}[\ln p(\mathbf {Z} ,\mathbf {X} )]+{\text{constant}}}">

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<mtext>constant</mtext>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ln q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )=\operatorname {E} _{i\neq j}[\ln p(\mathbf {Z} ,\mathbf {X} )]+{\text{constant}}}</annotation>

</semantics>

</math><img src="./931ba9fd5d59d6d74dad8cf2a82448fbca0c5ff9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.338ex; width:43.001ex; height:3.343ex;" alt="{\displaystyle \ln q_{j}^{*}(\mathbf {Z} _{j}\mid \mathbf {X} )=\operatorname {E} _{i\neq j}[\ln p(\mathbf {Z} ,\mathbf {X} )]+{\text{constant}}}" loading="lazy"> .</li>

<li>Fill in the formula for the <a href="Joint_probability_distribution" title="Joint probability distribution">joint probability distribution</a> using the graphical model. Any component conditional distributions that don't involve any of the variables in <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} _{j}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} _{j}}</annotation>

</semantics>

<li>Simplify the formula and apply the expectation operator, following the above example. Ideally, this should simplify into expectations of basic functions of variables not in <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} _{j}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} _{j}}</annotation>

</semantics>

<li>The functional form of the formula with respect to the variables in the current partition indicates the type of distribution. In particular, exponentiating the formula generates the <a href="Probability_density_function" title="Probability density function">probability density function</a> (PDF) of the distribution (or at least, something proportional to it, with unknown <a href="Normalization_constant" class="mw-redirect" title="Normalization constant">normalization constant</a>). In order for the overall method to be tractable, it should be possible to recognize the functional form as belonging to a known distribution. Significant mathematical manipulation may be required to convert the formula into a form that matches the PDF of a known distribution. When this can be done, the normalization constant can be reinstated by definition, and equations for the parameters of the known distribution can be derived by extracting the appropriate parts of the formula.</li>

<li>When all expectations can be replaced analytically with functions of variables not in the current partition, and the PDF put into a form that allows identification with a known distribution, the result is a set of equations expressing the values of the optimum parameters as functions of the parameters of variables in other partitions.</li>

<li>When this procedure can be applied to all partitions, the result is a set of mutually linked equations specifying the optimum values of all parameters.</li>

<li>An <a href="Expectation%E2%80%93maximization_algorithm" title="Expectation–maximization algorithm">expectation–maximization</a> (EM) type procedure is then applied, picking an initial value for each parameter and the iterating through a series of steps, where at each step we cycle through the equations, updating each parameter in turn. This is guaranteed to converge.</li></ol>

<div class="mw-heading mw-heading3"><h3 id="Most_important_points">Most important points</h3></div>

Due to all of the mathematical manipulations involved, it is easy to lose track of the big picture. The important things are:

<ol><li>The idea of variational Bayes is to construct an analytical approximation to the <a href="Posterior_probability" title="Posterior probability">posterior probability</a> of the set of unobserved variables (parameters and latent variables), given the data. This means that the form of the solution is similar to other <a href="Bayesian_inference" title="Bayesian inference">Bayesian inference</a> methods, such as <a href="Gibbs_sampling" title="Gibbs sampling">Gibbs sampling</a> — i.e. a distribution that seeks to describe everything that is known about the variables. As in other Bayesian methods — but unlike e.g. in <a href="Expectation%E2%80%93maximization_algorithm" title="Expectation–maximization algorithm">expectation–maximization</a> (EM) or other <a href="Maximum_likelihood" class="mw-redirect" title="Maximum likelihood">maximum likelihood</a> methods — both types of unobserved variables (i.e. parameters and latent variables) are treated the same, i.e. as <a href="Random_variable" title="Random variable">random variables</a>. Estimates for the variables can then be derived in the standard Bayesian ways, e.g. calculating the mean of the distribution to get a single point estimate or deriving a <a href="Credible_interval" title="Credible interval">credible interval</a>, highest density region, etc.</li>

<li>"Analytical approximation" means that a formula can be written down for the posterior distribution. The formula generally consists of a product of well-known probability distributions, each of which factorizes over a set of unobserved variables (i.e. it is <a href="Conditionally_independent" class="mw-redirect" title="Conditionally independent">conditionally independent</a> of the other variables, given the observed data). This formula is not the true posterior distribution, but an approximation to it; in particular, it will generally agree fairly closely in the lowest <a href="Moment_(mathematics)" title="Moment (mathematics)">moments</a> of the unobserved variables, e.g. the <a href="Mean" title="Mean">mean</a> and <a href="Variance" title="Variance">variance</a>.</li>

<li>The result of all of the mathematical manipulations is (1) the identity of the probability distributions making up the factors, and (2) mutually dependent formulas for the parameters of these distributions. The actual values of these parameters are computed numerically, through an alternating iterative procedure much like EM.</li></ol>

<div class="mw-heading mw-heading3"><h3 id="Compared_with_expectation–maximization_(EM)">Compared with expectation–maximization (EM)</h3></div>

Variational Bayes (VB) is often compared with <a href="Expectation%E2%80%93maximization_algorithm" title="Expectation–maximization algorithm">expectation–maximization</a> (EM). The actual numerical procedure is quite similar, in that both are alternating iterative procedures that successively converge on optimum parameter values. The initial steps to derive the respective procedures are also vaguely similar, both starting out with formulas for probability densities and both involving significant amounts of mathematical manipulations.

However, there are a number of differences. Most important is what is being computed.

<ul><li>EM computes point estimates of posterior distribution of those random variables that can be categorized as "parameters", but only estimates of the actual posterior distributions of the latent variables (at least in "soft EM", and often only when the latent variables are discrete). The point estimates computed are the <a href="Mode_(statistics)" title="Mode (statistics)">modes</a> of these parameters; no other information is available.</li>

<li>VB, on the other hand, computes estimates of the actual posterior distribution of all variables, both parameters and latent variables. When point estimates need to be derived, generally the <a href="Mean" title="Mean">mean</a> is used rather than the mode, as is normal in Bayesian inference. Concomitant with this, the parameters computed in VB do not have the same significance as those in EM. EM computes optimum values of the parameters of the Bayes network itself. VB computes optimum values of the parameters of the distributions used to approximate the parameters and latent variables of the Bayes network. For example, a typical Gaussian <a href="Mixture_model" title="Mixture model">mixture model</a> will have parameters for the mean and variance of each of the mixture components. EM would directly estimate optimum values for these parameters. VB, however, would first fit a distribution to these parameters — typically in the form of a <a href="Prior_distribution" class="mw-redirect" title="Prior distribution">prior distribution</a>, e.g. a <a href="Normal-scaled_inverse_gamma_distribution" class="mw-redirect" title="Normal-scaled inverse gamma distribution">normal-scaled inverse gamma distribution</a> — and would then compute values for the parameters of this prior distribution, i.e. essentially <a href="Hyperparameter_(Bayesian_statistics)" title="Hyperparameter (Bayesian statistics)">hyperparameters</a>. In this case, VB would compute optimum estimates of the four parameters of the normal-scaled inverse gamma distribution that describes the joint distribution of the mean and variance of the component.</li></ul>

<div class="mw-heading mw-heading2"><h2 id="A_more_complex_example">A more complex example</h2></div>

Imagine a Bayesian <a href="Gaussian_mixture_model" class="mw-redirect" title="Gaussian mixture model">Gaussian mixture model</a> described as follows:<a href="#cite_note-Nguyen-3">[3]</a>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

<mi>SymDir</mi>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msub>

<msup>

<msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msub>

</mrow>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

<mtext>number of mixing components</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

<mtext>number of data points</mtext>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\mathbf {\pi } &\sim \operatorname {SymDir} (K,\alpha _{0})\\\mathbf {\Lambda } _{i=1\dots K}&\sim {\mathcal {W}}(\mathbf {W} _{0},\nu _{0})\\\mathbf {\mu } _{i=1\dots K}&\sim {\mathcal {N}}(\mathbf {\mu } _{0},(\beta _{0}\mathbf {\Lambda } _{i})^{-1})\\\mathbf {z} [i=1\dots N]&\sim \operatorname {Mult} (1,\mathbf {\pi } )\\\mathbf {x} _{i=1\dots N}&\sim {\mathcal {N}}(\mathbf {\mu } _{z_{i}},{\mathbf {\Lambda } _{z_{i}}}^{-1})\\K&={\text{number of mixing components}}\\N&={\text{number of data points}}\end{aligned}}}</annotation>

</semantics>

</math><img src="./6e54f90a522209bd5e80751661fbd8f25570b952.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -10.671ex; width:47.266ex; height:22.509ex;" alt="{\displaystyle {\begin{aligned}\mathbf {\pi } &\sim \operatorname {SymDir} (K,\alpha _{0})\\\mathbf {\Lambda } _{i=1\dots K}&\sim {\mathcal {W}}(\mathbf {W} _{0},\nu _{0})\\\mathbf {\mu } _{i=1\dots K}&\sim {\mathcal {N}}(\mathbf {\mu } _{0},(\beta _{0}\mathbf {\Lambda } _{i})^{-1})\\\mathbf {z} [i=1\dots N]&\sim \operatorname {Mult} (1,\mathbf {\pi } )\\\mathbf {x} _{i=1\dots N}&\sim {\mathcal {N}}(\mathbf {\mu } _{z_{i}},{\mathbf {\Lambda } _{z_{i}}}^{-1})\\K&={\text{number of mixing components}}\\N&={\text{number of data points}}\end{aligned}}}" loading="lazy"></dd></dl>

Note:

<ul><li>SymDir() is the symmetric <a href="Dirichlet_distribution" title="Dirichlet distribution">Dirichlet distribution</a> of dimension <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle K}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle K}</annotation>

</semantics>

</math><img src="./2b76fce82a62ed5461908f0dc8f037de4e3686b0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.066ex; height:2.176ex;" alt="{\displaystyle K}" loading="lazy">, with the hyperparameter for each component set to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \alpha _{0}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \alpha _{0}}</annotation>

</semantics>

</math><img src="./a214eff2fcc322f780dd8837e7472b0edb994a13.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.542ex; height:2.009ex;" alt="{\displaystyle \alpha _{0}}" loading="lazy">. The Dirichlet distribution is the <a href="Conjugate_prior" title="Conjugate prior">conjugate prior</a> of the <a href="Categorical_distribution" title="Categorical distribution">categorical distribution</a> or <a href="Multinomial_distribution" title="Multinomial distribution">multinomial distribution</a>.</li>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {W}}()}</annotation>

</semantics>

</math><img src="./ea721c03e0cad4fc58cdb4e60c2881b96677a27c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.214ex; height:2.843ex;" alt="{\displaystyle {\mathcal {W}}()}" loading="lazy"> is the <a href="Wishart_distribution" title="Wishart distribution">Wishart distribution</a>, which is the conjugate prior of the <a href="Precision_matrix" class="mw-redirect" title="Precision matrix">precision matrix</a> (inverse <a href="Covariance_matrix" title="Covariance matrix">covariance matrix</a>) for a <a href="Multivariate_Gaussian_distribution" class="mw-redirect" title="Multivariate Gaussian distribution">multivariate Gaussian distribution</a>.</li>

<li>Mult() is a <a href="Multinomial_distribution" title="Multinomial distribution">multinomial distribution</a> over a single observation (equivalent to a <a href="Categorical_distribution" title="Categorical distribution">categorical distribution</a>). The state space is a "one-of-K" representation, i.e., a <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle K}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle K}</annotation>

</semantics>

</math><img src="./2b76fce82a62ed5461908f0dc8f037de4e3686b0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.066ex; height:2.176ex;" alt="{\displaystyle K}" loading="lazy">-dimensional vector in which one of the elements is 1 (specifying the identity of the observation) and all other elements are 0.</li>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {N}}()}</annotation>

</semantics>

</math><img src="./dcc84cb16dc05a84b640ffb5f96cce22af8fa0b8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.062ex; width:4.146ex; height:3.009ex;" alt="{\displaystyle {\mathcal {N}}()}" loading="lazy"> is the <a href="Gaussian_distribution" class="mw-redirect" title="Gaussian distribution">Gaussian distribution</a>, in this case specifically the <a href="Multivariate_Gaussian_distribution" class="mw-redirect" title="Multivariate Gaussian distribution">multivariate Gaussian distribution</a>.</li></ul>

The interpretation of the above variables is as follows:

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {X} =\{\mathbf {x} _{1},\dots ,\mathbf {x} _{N}\}}</annotation>

</semantics>

</math><img src="./899f889b086f2e9b956c3e937ce6793f9cb66faa.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:18.189ex; height:2.843ex;" alt="{\displaystyle \mathbf {X} =\{\mathbf {x} _{1},\dots ,\mathbf {x} _{N}\}}" loading="lazy"> is the set of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle N}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle N}</annotation>

</semantics>

</math><img src="./f5e3890c981ae85503089652feb48b191b57aae3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.064ex; height:2.176ex;" alt="{\displaystyle N}" loading="lazy"> data points, each of which is a <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle D}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle D}</annotation>

</semantics>

</math><img src="./f34a0c600395e5d4345287e21fb26efd386990e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.924ex; height:2.176ex;" alt="{\displaystyle D}" loading="lazy">-dimensional vector distributed according to a <a href="Multivariate_Gaussian_distribution" class="mw-redirect" title="Multivariate Gaussian distribution">multivariate Gaussian distribution</a>.</li>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} =\{\mathbf {z} _{1},\dots ,\mathbf {z} _{N}\}}</annotation>

</semantics>

</math><img src="./5959d0417f5c451bf7d812d5bc2d2a16854c12e9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:17.357ex; height:2.843ex;" alt="{\displaystyle \mathbf {Z} =\{\mathbf {z} _{1},\dots ,\mathbf {z} _{N}\}}" loading="lazy"> is a set of latent variables, one per data point, specifying which mixture component the corresponding data point belongs to, using a "one-of-K" vector representation with components <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle z_{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle z_{nk}}</annotation>

</semantics>

</math><img src="./50c1a2ebeb33c48ba0e8ba03176918425862d4c9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.156ex; height:2.009ex;" alt="{\displaystyle z_{nk}}" loading="lazy"> for <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle k=1\dots K}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle k=1\dots K}</annotation>

</semantics>

</math><img src="./f1b2624be134bda7a383bf912a040d6579a6127c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:11.036ex; height:2.176ex;" alt="{\displaystyle k=1\dots K}" loading="lazy">, as described above.</li>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {\pi } }</annotation>

</semantics>

</math><img src="./07b0cee10b7be89207fd2de25f81a23f11018ff8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.332ex; height:1.676ex;" alt="{\displaystyle \mathbf {\pi } }" loading="lazy"> is the mixing proportions for the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle K}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle K}</annotation>

</semantics>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {\mu } _{i=1\dots K}}</annotation>

</semantics>

</math><img src="./89b56734561b9a53296b3c7abe3463e2798e9ac7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:7.688ex; height:2.176ex;" alt="{\displaystyle \mathbf {\mu } _{i=1\dots K}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {\Lambda } _{i=1\dots K}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {\Lambda } _{i=1\dots K}}</annotation>

</semantics>

</math><img src="./8397707df58b255aeabec6047b0d01f5140241e2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:8.16ex; height:2.509ex;" alt="{\displaystyle \mathbf {\Lambda } _{i=1\dots K}}" loading="lazy"> specify the parameters (<a href="Mean" title="Mean">mean</a> and <a href="Precision_(statistics)" title="Precision (statistics)">precision</a>) associated with each mixture component.</li></ul>

The joint probability of all variables can be rewritten as

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p(\mathbf {X} ,\mathbf {Z} ,\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )=p(\mathbf {X} \mid \mathbf {Z} ,\mathbf {\mu } ,\mathbf {\Lambda } )p(\mathbf {Z} \mid \mathbf {\pi } )p(\mathbf {\pi } )p(\mathbf {\mu } \mid \mathbf {\Lambda } )p(\mathbf {\Lambda } )}</annotation>

</semantics>

</math><img src="./c583236ba92e993d0a8bbcd77ea2afde26f4ec14.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.089ex; width:57.71ex; height:2.843ex;" alt="{\displaystyle p(\mathbf {X} ,\mathbf {Z} ,\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )=p(\mathbf {X} \mid \mathbf {Z} ,\mathbf {\mu } ,\mathbf {\Lambda } )p(\mathbf {Z} \mid \mathbf {\pi } )p(\mathbf {\pi } )p(\mathbf {\mu } \mid \mathbf {\Lambda } )p(\mathbf {\Lambda } )}" loading="lazy"></dd></dl>

where the individual factors are

<mtr>

<mtd>

</mrow>

</mrow>

</mrow>

</mrow>

</mtd>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msubsup>

<msup>

<msub>

</mrow>

</msub>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</mtd>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</mtd>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}p(\mathbf {X} \mid \mathbf {Z} ,\mathbf {\mu } ,\mathbf {\Lambda } )&=\prod _{n=1}^{N}\prod _{k=1}^{K}{\mathcal {N}}(\mathbf {x} _{n}\mid \mathbf {\mu } _{k},\mathbf {\Lambda } _{k}^{-1})^{z_{nk}}\\p(\mathbf {Z} \mid \mathbf {\pi } )&=\prod _{n=1}^{N}\prod _{k=1}^{K}\pi _{k}^{z_{nk}}\\p(\mathbf {\pi } )&={\frac {\Gamma (K\alpha _{0})}{\Gamma (\alpha _{0})^{K}}}\prod _{k=1}^{K}\pi _{k}^{\alpha _{0}-1}\\p(\mathbf {\mu } \mid \mathbf {\Lambda } )&=\prod _{k=1}^{K}{\mathcal {N}}(\mathbf {\mu } _{k}\mid \mathbf {\mu } _{0},(\beta _{0}\mathbf {\Lambda } _{k})^{-1})\\p(\mathbf {\Lambda } )&=\prod _{k=1}^{K}{\mathcal {W}}(\mathbf {\Lambda } _{k}\mid \mathbf {W} _{0},\nu _{0})\end{aligned}}}</annotation>

</semantics>

</math><img src="./4fb8a250bfc8ff4c25d41fe537d73505462fe5c7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -17.955ex; margin-bottom: -0.216ex; width:43.631ex; height:37.509ex;" alt="{\displaystyle {\begin{aligned}p(\mathbf {X} \mid \mathbf {Z} ,\mathbf {\mu } ,\mathbf {\Lambda } )&=\prod _{n=1}^{N}\prod _{k=1}^{K}{\mathcal {N}}(\mathbf {x} _{n}\mid \mathbf {\mu } _{k},\mathbf {\Lambda } _{k}^{-1})^{z_{nk}}\\p(\mathbf {Z} \mid \mathbf {\pi } )&=\prod _{n=1}^{N}\prod _{k=1}^{K}\pi _{k}^{z_{nk}}\\p(\mathbf {\pi } )&={\frac {\Gamma (K\alpha _{0})}{\Gamma (\alpha _{0})^{K}}}\prod _{k=1}^{K}\pi _{k}^{\alpha _{0}-1}\\p(\mathbf {\mu } \mid \mathbf {\Lambda } )&=\prod _{k=1}^{K}{\mathcal {N}}(\mathbf {\mu } _{k}\mid \mathbf {\mu } _{0},(\beta _{0}\mathbf {\Lambda } _{k})^{-1})\\p(\mathbf {\Lambda } )&=\prod _{k=1}^{K}{\mathcal {W}}(\mathbf {\Lambda } _{k}\mid \mathbf {W} _{0},\nu _{0})\end{aligned}}}" loading="lazy"></dd></dl>

where

<mtr>

<mtd>

</mrow>

</mrow>

</mrow>

</mrow>

</mtd>

<mtd>

<mfrac>

<mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

<mfrac>

<mrow>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</mrow>

</mtd>

<mtd>

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</mrow>

</msup>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</mrow>

</msup>

<msup>

<mrow>

<mrow>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</munderover>

<mrow>

<mfrac>

<mrow>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

<mtext>dimensionality of each data point</mtext>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}{\mathcal {N}}(\mathbf {x} \mid \mathbf {\mu } ,\mathbf {\Sigma } )&={\frac {1}{(2\pi )^{D/2}}}{\frac {1}{|\mathbf {\Sigma } |^{1/2}}}\exp \left\{-{\frac {1}{2}}(\mathbf {x} -\mathbf {\mu } )^{\rm {T}}\mathbf {\Sigma } ^{-1}(\mathbf {x} -\mathbf {\mu } )\right\}\\{\mathcal {W}}(\mathbf {\Lambda } \mid \mathbf {W} ,\nu )&=B(\mathbf {W} ,\nu )|\mathbf {\Lambda } |^{(\nu -D-1)/2}\exp \left(-{\frac {1}{2}}\operatorname {Tr} (\mathbf {W} ^{-1}\mathbf {\Lambda } )\right)\\B(\mathbf {W} ,\nu )&=|\mathbf {W} |^{-\nu /2}\left\{2^{\nu D/2}\pi ^{D(D-1)/4}\prod _{i=1}^{D}\Gamma \left({\frac {\nu +1-i}{2}}\right)\right\}^{-1}\\D&={\text{dimensionality of each data point}}\end{aligned}}}</annotation>

</semantics>

</math><img src="./6e297b20d7c9af755fdc79dfe634cc58ccc4c122.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -11.505ex; width:64.366ex; height:24.176ex;" alt="{\displaystyle {\begin{aligned}{\mathcal {N}}(\mathbf {x} \mid \mathbf {\mu } ,\mathbf {\Sigma } )&={\frac {1}{(2\pi )^{D/2}}}{\frac {1}{|\mathbf {\Sigma } |^{1/2}}}\exp \left\{-{\frac {1}{2}}(\mathbf {x} -\mathbf {\mu } )^{\rm {T}}\mathbf {\Sigma } ^{-1}(\mathbf {x} -\mathbf {\mu } )\right\}\\{\mathcal {W}}(\mathbf {\Lambda } \mid \mathbf {W} ,\nu )&=B(\mathbf {W} ,\nu )|\mathbf {\Lambda } |^{(\nu -D-1)/2}\exp \left(-{\frac {1}{2}}\operatorname {Tr} (\mathbf {W} ^{-1}\mathbf {\Lambda } )\right)\\B(\mathbf {W} ,\nu )&=|\mathbf {W} |^{-\nu /2}\left\{2^{\nu D/2}\pi ^{D(D-1)/4}\prod _{i=1}^{D}\Gamma \left({\frac {\nu +1-i}{2}}\right)\right\}^{-1}\\D&={\text{dimensionality of each data point}}\end{aligned}}}" loading="lazy"></dd></dl>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q(\mathbf {Z} ,\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )=q(\mathbf {Z} )q(\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )}</annotation>

</semantics>

</math><img src="./48318f9eaa5512ae1b1432aeabe75506c837713d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:29.386ex; height:2.843ex;" alt="{\displaystyle q(\mathbf {Z} ,\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )=q(\mathbf {Z} )q(\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )}" loading="lazy">.

Then<a href="#cite_note-Nguyen-3">[3]</a>

<mtr>

<mtd>

<msup>

</mrow>

</msup>

</mrow>

</mtd>

<mtd>

<msub>

</mrow>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

<mtext>constant</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</mrow>

<mtext>constant</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<mtext>constant</mtext>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\ln q^{*}(\mathbf {Z} )&=\operatorname {E} _{\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } }[\ln p(\mathbf {X} ,\mathbf {Z} ,\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )]+{\text{constant}}\\&=\operatorname {E} _{\mathbf {\pi } }[\ln p(\mathbf {Z} \mid \mathbf {\pi } )]+\operatorname {E} _{\mathbf {\mu } ,\mathbf {\Lambda } }[\ln p(\mathbf {X} \mid \mathbf {Z} ,\mathbf {\mu } ,\mathbf {\Lambda } )]+{\text{constant}}\\&=\sum _{n=1}^{N}\sum _{k=1}^{K}z_{nk}\ln \rho _{nk}+{\text{constant}}\end{aligned}}}</annotation>

</semantics>

</math><img src="./a9e6956e240ed7d5d347063c687111ca758eabfd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -6.338ex; width:62.481ex; height:13.843ex;" alt="{\displaystyle {\begin{aligned}\ln q^{*}(\mathbf {Z} )&=\operatorname {E} _{\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } }[\ln p(\mathbf {X} ,\mathbf {Z} ,\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )]+{\text{constant}}\\&=\operatorname {E} _{\mathbf {\pi } }[\ln p(\mathbf {Z} \mid \mathbf {\pi } )]+\operatorname {E} _{\mathbf {\mu } ,\mathbf {\Lambda } }[\ln p(\mathbf {X} \mid \mathbf {Z} ,\mathbf {\mu } ,\mathbf {\Lambda } )]+{\text{constant}}\\&=\sum _{n=1}^{N}\sum _{k=1}^{K}z_{nk}\ln \rho _{nk}+{\text{constant}}\end{aligned}}}" loading="lazy"></dd></dl>

where we have defined

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

<mfrac>

</mfrac>

</mrow>

<mfrac>

</mfrac>

</mrow>

<msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ln \rho _{nk}=\operatorname {E} [\ln \pi _{k}]+{\frac {1}{2}}\operatorname {E} [\ln |\mathbf {\Lambda } _{k}|]-{\frac {D}{2}}\ln(2\pi )-{\frac {1}{2}}\operatorname {E} _{\mathbf {\mu } _{k},\mathbf {\Lambda } _{k}}[(\mathbf {x} _{n}-\mathbf {\mu } _{k})^{\rm {T}}\mathbf {\Lambda } _{k}(\mathbf {x} _{n}-\mathbf {\mu } _{k})]}</annotation>

</semantics>

</math><img src="./6ec0788f8a16fa7e700c96d3af8382f54cbb287f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.838ex; width:79.981ex; height:5.176ex;" alt="{\displaystyle \ln \rho _{nk}=\operatorname {E} [\ln \pi _{k}]+{\frac {1}{2}}\operatorname {E} [\ln |\mathbf {\Lambda } _{k}|]-{\frac {D}{2}}\ln(2\pi )-{\frac {1}{2}}\operatorname {E} _{\mathbf {\mu } _{k},\mathbf {\Lambda } _{k}}[(\mathbf {x} _{n}-\mathbf {\mu } _{k})^{\rm {T}}\mathbf {\Lambda } _{k}(\mathbf {x} _{n}-\mathbf {\mu } _{k})]}" loading="lazy"></dd></dl>

Exponentiating both sides of the formula for <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ln q^{*}(\mathbf {Z} )}">

<msup>

</mrow>

</msup>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ln q^{*}(\mathbf {Z} )}</annotation>

</semantics>

</math><img src="./0b004a9635c4925745f96e7f15050926fc4d78bd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:7.903ex; height:2.843ex;" alt="{\displaystyle \ln q^{*}(\mathbf {Z} )}" loading="lazy"> yields

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{*}(\mathbf {Z} )\propto \prod _{n=1}^{N}\prod _{k=1}^{K}\rho _{nk}^{z_{nk}}}</annotation>

</semantics>

</math><img src="./dea44fe4e56df5e195593dbe748ac45b9dccaa35.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:19.365ex; height:7.343ex;" alt="{\displaystyle q^{*}(\mathbf {Z} )\propto \prod _{n=1}^{N}\prod _{k=1}^{K}\rho _{nk}^{z_{nk}}}" loading="lazy"></dd></dl>

Requiring that this be normalized ends up requiring that the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \rho _{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \rho _{nk}}</annotation>

</semantics>

</math><img src="./e8adde11d7a36e6ac15d6ff18a40dc68ca5c5861.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:3.277ex; height:2.176ex;" alt="{\displaystyle \rho _{nk}}" loading="lazy"> sum to 1 over all values of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle k}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle k}</annotation>

</semantics>

</math><img src="./c3c9a2c7b599b37105512c5d570edc034056dd40.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.211ex; height:2.176ex;" alt="{\displaystyle k}" loading="lazy">, yielding

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{*}(\mathbf {Z} )=\prod _{n=1}^{N}\prod _{k=1}^{K}r_{nk}^{z_{nk}}}</annotation>

</semantics>

</math><img src="./76a0934ae8df172e6ba425f89ffe383f5367fb34.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:19.211ex; height:7.343ex;" alt="{\displaystyle q^{*}(\mathbf {Z} )=\prod _{n=1}^{N}\prod _{k=1}^{K}r_{nk}^{z_{nk}}}" loading="lazy"></dd></dl>

where

<msub>

</mrow>

</msub>

<mfrac>

<msub>

</mrow>

</msub>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle r_{nk}={\frac {\rho _{nk}}{\sum _{j=1}^{K}\rho _{nj}}}}</annotation>

</semantics>

</math><img src="./c142dc2891f560561d42c3630bd402e3436c9e48.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.505ex; width:16.007ex; height:6.509ex;" alt="{\displaystyle r_{nk}={\frac {\rho _{nk}}{\sum _{j=1}^{K}\rho _{nj}}}}" loading="lazy"></dd></dl>

In other words, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q^{*}(\mathbf {Z} )}">

<msup>

</mrow>

</msup>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{*}(\mathbf {Z} )}</annotation>

</semantics>

</math><img src="./90de3cc99b18da7ee87f98d0cd7cd66b7c791836.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.577ex; height:2.843ex;" alt="{\displaystyle q^{*}(\mathbf {Z} )}" loading="lazy"> is a product of single-observation <a href="Multinomial_distribution" title="Multinomial distribution">multinomial distributions</a>, and factors over each individual <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {z} _{n}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {z} _{n}}</annotation>

</semantics>

</math><img src="./ff3c4e54a2f18ad992a5eb8057055d92e1389f5f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.406ex; height:2.009ex;" alt="{\displaystyle \mathbf {z} _{n}}" loading="lazy">, which is distributed as a single-observation multinomial distribution with parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle r_{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle r_{nk}}</annotation>

</semantics>

</math><img src="./d2f640f5e142f2e7b490500433a81e863a04413a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.124ex; height:2.009ex;" alt="{\displaystyle r_{nk}}" loading="lazy"> for <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle k=1\dots K}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle k=1\dots K}</annotation>

</semantics>

Furthermore, we note that

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {E} [z_{nk}]=r_{nk}\,}</annotation>

</semantics>

</math><img src="./c15f150fa73aec952164769cfde73f6df58590e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:12.642ex; height:2.843ex;" alt="{\displaystyle \operatorname {E} [z_{nk}]=r_{nk}\,}" loading="lazy"></dd></dl>

which is a standard result for categorical distributions.

Now, considering the factor <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q(\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )}">

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q(\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )}</annotation>

</semantics>

</math><img src="./475da78c442f8bf3b83fbadb83249702f9e3517f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.554ex; height:2.843ex;" alt="{\displaystyle q(\mathbf {\pi } ,\mathbf {\mu } ,\mathbf {\Lambda } )}" loading="lazy">, note that it automatically factors into <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q(\mathbf {\pi } )\prod _{k=1}^{K}q(\mathbf {\mu } _{k},\mathbf {\Lambda } _{k})}">

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q(\mathbf {\pi } )\prod _{k=1}^{K}q(\mathbf {\mu } _{k},\mathbf {\Lambda } _{k})}</annotation>

</semantics>

</math><img src="./89d2a7f45d1252065f836fb12ef31675e9705733.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:17.32ex; height:7.343ex;" alt="{\displaystyle q(\mathbf {\pi } )\prod _{k=1}^{K}q(\mathbf {\mu } _{k},\mathbf {\Lambda } _{k})}" loading="lazy"> due to the structure of the graphical model defining our Gaussian mixture model, which is specified above.

Then,

<mtr>

<mtd>

<msup>

</mrow>

</msup>

</mrow>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<mtext>constant</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<mtext>constant</mtext>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\ln q^{*}(\mathbf {\pi } )&=\ln p(\mathbf {\pi } )+\operatorname {E} _{\mathbf {Z} }[\ln p(\mathbf {Z} \mid \mathbf {\pi } )]+{\text{constant}}\\&=(\alpha _{0}-1)\sum _{k=1}^{K}\ln \pi _{k}+\sum _{n=1}^{N}\sum _{k=1}^{K}r_{nk}\ln \pi _{k}+{\text{constant}}\end{aligned}}}</annotation>

</semantics>

</math><img src="./9ce937b397d56eb7f4ee329522f5ba5379abd69f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -4.671ex; width:58.759ex; height:10.509ex;" alt="{\displaystyle {\begin{aligned}\ln q^{*}(\mathbf {\pi } )&=\ln p(\mathbf {\pi } )+\operatorname {E} _{\mathbf {Z} }[\ln p(\mathbf {Z} \mid \mathbf {\pi } )]+{\text{constant}}\\&=(\alpha _{0}-1)\sum _{k=1}^{K}\ln \pi _{k}+\sum _{n=1}^{N}\sum _{k=1}^{K}r_{nk}\ln \pi _{k}+{\text{constant}}\end{aligned}}}" loading="lazy"></dd></dl>

Taking the exponential of both sides, we recognize <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q^{*}(\mathbf {\pi } )}">

<msup>

</mrow>

</msup>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{*}(\mathbf {\pi } )}</annotation>

</semantics>

</math><img src="./9aa11c5f81ece5855102329edd0d414cb9b2ddc7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.275ex; height:2.843ex;" alt="{\displaystyle q^{*}(\mathbf {\pi } )}" loading="lazy"> as a <a href="Dirichlet_distribution" title="Dirichlet distribution">Dirichlet distribution</a>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{*}(\mathbf {\pi } )\sim \operatorname {Dir} (\mathbf {\alpha } )\,}</annotation>

</semantics>

</math><img src="./7746f0edeceda0c8599bf54a318591737792fed4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:15.392ex; height:2.843ex;" alt="{\displaystyle q^{*}(\mathbf {\pi } )\sim \operatorname {Dir} (\mathbf {\alpha } )\,}" loading="lazy"></dd></dl>

where

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \alpha _{k}=\alpha _{0}+N_{k}\,}</annotation>

</semantics>

</math><img src="./cc05ba94245d5aa284388af9eb42fcc16603818d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:14.399ex; height:2.509ex;" alt="{\displaystyle \alpha _{k}=\alpha _{0}+N_{k}\,}" loading="lazy"></dd></dl>

where

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle N_{k}=\sum _{n=1}^{N}r_{nk}\,}</annotation>

</semantics>

</math><img src="./576dd9a643f447e3f861754be8595ebd728d3dcd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:13.306ex; height:7.343ex;" alt="{\displaystyle N_{k}=\sum _{n=1}^{N}r_{nk}\,}" loading="lazy"></dd></dl>

Finally

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msubsup>

<mtext>constant</mtext>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ln q^{*}(\mathbf {\mu } _{k},\mathbf {\Lambda } _{k})=\ln p(\mathbf {\mu } _{k},\mathbf {\Lambda } _{k})+\sum _{n=1}^{N}\operatorname {E} [z_{nk}]\ln {\mathcal {N}}(\mathbf {x} _{n}\mid \mathbf {\mu } _{k},\mathbf {\Lambda } _{k}^{-1})+{\text{constant}}}</annotation>

</semantics>

</math><img src="./cc21b5385915806fac7030e8de06621c21ee716c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:70.864ex; height:7.343ex;" alt="{\displaystyle \ln q^{*}(\mathbf {\mu } _{k},\mathbf {\Lambda } _{k})=\ln p(\mathbf {\mu } _{k},\mathbf {\Lambda } _{k})+\sum _{n=1}^{N}\operatorname {E} [z_{nk}]\ln {\mathcal {N}}(\mathbf {x} _{n}\mid \mathbf {\mu } _{k},\mathbf {\Lambda } _{k}^{-1})+{\text{constant}}}" loading="lazy"></dd></dl>

Grouping and reading off terms involving <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {\mu } _{k}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {\mu } _{k}}</annotation>

</semantics>

</math><img src="./bbc0e86cd6298aec58f95ff25c7171dc10e330be.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:2.49ex; height:2.176ex;" alt="{\displaystyle \mathbf {\mu } _{k}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {\Lambda } _{k}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {\Lambda } _{k}}</annotation>

</semantics>

</math><img src="./969af1cce7839ff781158a8e64e5643ca85e4f27.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.962ex; height:2.509ex;" alt="{\displaystyle \mathbf {\Lambda } _{k}}" loading="lazy">, the result is a <a href="Gaussian-Wishart_distribution" class="mw-redirect" title="Gaussian-Wishart distribution">Gaussian-Wishart distribution</a> given by

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{*}(\mathbf {\mu } _{k},\mathbf {\Lambda } _{k})={\mathcal {N}}(\mathbf {\mu } _{k}\mid \mathbf {m} _{k},(\beta _{k}\mathbf {\Lambda } _{k})^{-1}){\mathcal {W}}(\mathbf {\Lambda } _{k}\mid \mathbf {W} _{k},\nu _{k})}</annotation>

</semantics>

</math><img src="./30dc879001e5185ffd06cc5441babfc3c04e8fce.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:52.133ex; height:3.176ex;" alt="{\displaystyle q^{*}(\mathbf {\mu } _{k},\mathbf {\Lambda } _{k})={\mathcal {N}}(\mathbf {\mu } _{k}\mid \mathbf {m} _{k},(\beta _{k}\mathbf {\Lambda } _{k})^{-1}){\mathcal {W}}(\mathbf {\Lambda } _{k}\mid \mathbf {W} _{k},\nu _{k})}" loading="lazy"></dd></dl>

given the definitions

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

<mfrac>

<msub>

</mrow>

</msub>

</mfrac>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

<mover>

</mrow>

</mover>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

</mrow>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<msub>

<mover>

</mrow>

</mover>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

<mover>

</mrow>

</mover>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

<mover>

</mrow>

</mover>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

<mfrac>

<msub>

</mrow>

</msub>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

<mfrac>

<msub>

</mrow>

</msub>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

<mover>

</mrow>

</mover>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

<mover>

</mrow>

</mover>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\beta _{k}&=\beta _{0}+N_{k}\\\mathbf {m} _{k}&={\frac {1}{\beta _{k}}}(\beta _{0}\mathbf {\mu } _{0}+N_{k}{\bar {\mathbf {x} }}_{k})\\\mathbf {W} _{k}^{-1}&=\mathbf {W} _{0}^{-1}+N_{k}\mathbf {S} _{k}+{\frac {\beta _{0}N_{k}}{\beta _{0}+N_{k}}}({\bar {\mathbf {x} }}_{k}-\mathbf {\mu } _{0})({\bar {\mathbf {x} }}_{k}-\mathbf {\mu } _{0})^{\rm {T}}\\\nu _{k}&=\nu _{0}+N_{k}\\N_{k}&=\sum _{n=1}^{N}r_{nk}\\{\bar {\mathbf {x} }}_{k}&={\frac {1}{N_{k}}}\sum _{n=1}^{N}r_{nk}\mathbf {x} _{n}\\\mathbf {S} _{k}&={\frac {1}{N_{k}}}\sum _{n=1}^{N}r_{nk}(\mathbf {x} _{n}-{\bar {\mathbf {x} }}_{k})(\mathbf {x} _{n}-{\bar {\mathbf {x} }}_{k})^{\rm {T}}\end{aligned}}}</annotation>

</semantics>

</math><img src="./a5e390b9fe838d3864e07b6fa538f3e4047605b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -19.396ex; margin-bottom: -0.275ex; width:54.882ex; height:40.509ex;" alt="{\displaystyle {\begin{aligned}\beta _{k}&=\beta _{0}+N_{k}\\\mathbf {m} _{k}&={\frac {1}{\beta _{k}}}(\beta _{0}\mathbf {\mu } _{0}+N_{k}{\bar {\mathbf {x} }}_{k})\\\mathbf {W} _{k}^{-1}&=\mathbf {W} _{0}^{-1}+N_{k}\mathbf {S} _{k}+{\frac {\beta _{0}N_{k}}{\beta _{0}+N_{k}}}({\bar {\mathbf {x} }}_{k}-\mathbf {\mu } _{0})({\bar {\mathbf {x} }}_{k}-\mathbf {\mu } _{0})^{\rm {T}}\\\nu _{k}&=\nu _{0}+N_{k}\\N_{k}&=\sum _{n=1}^{N}r_{nk}\\{\bar {\mathbf {x} }}_{k}&={\frac {1}{N_{k}}}\sum _{n=1}^{N}r_{nk}\mathbf {x} _{n}\\\mathbf {S} _{k}&={\frac {1}{N_{k}}}\sum _{n=1}^{N}r_{nk}(\mathbf {x} _{n}-{\bar {\mathbf {x} }}_{k})(\mathbf {x} _{n}-{\bar {\mathbf {x} }}_{k})^{\rm {T}}\end{aligned}}}" loading="lazy"></dd></dl>

Finally, notice that these functions require the values of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle r_{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle r_{nk}}</annotation>

</semantics>

</math><img src="./d2f640f5e142f2e7b490500433a81e863a04413a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.124ex; height:2.009ex;" alt="{\displaystyle r_{nk}}" loading="lazy">, which make use of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \rho _{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \rho _{nk}}</annotation>

</semantics>

</math><img src="./e8adde11d7a36e6ac15d6ff18a40dc68ca5c5861.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:3.277ex; height:2.176ex;" alt="{\displaystyle \rho _{nk}}" loading="lazy">, which is defined in turn based on <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \operatorname {E} [\ln \pi _{k}]}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {E} [\ln \pi _{k}]}</annotation>

</semantics>

</math><img src="./7038f8179432a3693f8296feacc5f8874ba6f267.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:7.617ex; height:2.843ex;" alt="{\displaystyle \operatorname {E} [\ln \pi _{k}]}" loading="lazy">, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \operatorname {E} [\ln |\mathbf {\Lambda } _{k}|]}">

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {E} [\ln |\mathbf {\Lambda } _{k}|]}</annotation>

</semantics>

</math><img src="./c07e1eecf8caee684700b0e4b63cc117051feab1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.459ex; height:2.843ex;" alt="{\displaystyle \operatorname {E} [\ln |\mathbf {\Lambda } _{k}|]}" loading="lazy">, and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \operatorname {E} _{\mathbf {\mu } _{k},\mathbf {\Lambda } _{k}}[(\mathbf {x} _{n}-\mathbf {\mu } _{k})^{\rm {T}}\mathbf {\Lambda } _{k}(\mathbf {x} _{n}-\mathbf {\mu } _{k})]}">

<msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {E} _{\mathbf {\mu } _{k},\mathbf {\Lambda } _{k}}[(\mathbf {x} _{n}-\mathbf {\mu } _{k})^{\rm {T}}\mathbf {\Lambda } _{k}(\mathbf {x} _{n}-\mathbf {\mu } _{k})]}</annotation>

</semantics>

</math><img src="./632ace27969c8d456caa65b802e8e8ce268e51bb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:31.521ex; height:3.343ex;" alt="{\displaystyle \operatorname {E} _{\mathbf {\mu } _{k},\mathbf {\Lambda } _{k}}[(\mathbf {x} _{n}-\mathbf {\mu } _{k})^{\rm {T}}\mathbf {\Lambda } _{k}(\mathbf {x} _{n}-\mathbf {\mu } _{k})]}" loading="lazy">. Now that we have determined the distributions over which these expectations are taken, we can derive formulas for them:

<mtr>

<mtd>

<msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</munderover>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

<mrow>

<mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\operatorname {E} _{\mathbf {\mu } _{k},\mathbf {\Lambda } _{k}}[(\mathbf {x} _{n}-\mathbf {\mu } _{k})^{\rm {T}}\mathbf {\Lambda } _{k}(\mathbf {x} _{n}-\mathbf {\mu } _{k})]&=D\beta _{k}^{-1}+\nu _{k}(\mathbf {x} _{n}-\mathbf {m} _{k})^{\rm {T}}\mathbf {W} _{k}(\mathbf {x} _{n}-\mathbf {m} _{k})\\\ln {\widetilde {\Lambda }}_{k}&\equiv \operatorname {E} [\ln |\mathbf {\Lambda } _{k}|]=\sum _{i=1}^{D}\psi \left({\frac {\nu _{k}+1-i}{2}}\right)+D\ln 2+\ln |\mathbf {W} _{k}|\\\ln {\widetilde {\pi }}_{k}&\equiv \operatorname {E} \left[\ln |\pi _{k}|\right]=\psi (\alpha _{k})-\psi \left(\sum _{i=1}^{K}\alpha _{i}\right)\end{aligned}}}</annotation>

</semantics>

</math><img src="./ef9311bd1332799fbb4578b2547cd94b56b37acd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -8.671ex; width:86.663ex; height:18.509ex;" alt="{\displaystyle {\begin{aligned}\operatorname {E} _{\mathbf {\mu } _{k},\mathbf {\Lambda } _{k}}[(\mathbf {x} _{n}-\mathbf {\mu } _{k})^{\rm {T}}\mathbf {\Lambda } _{k}(\mathbf {x} _{n}-\mathbf {\mu } _{k})]&=D\beta _{k}^{-1}+\nu _{k}(\mathbf {x} _{n}-\mathbf {m} _{k})^{\rm {T}}\mathbf {W} _{k}(\mathbf {x} _{n}-\mathbf {m} _{k})\\\ln {\widetilde {\Lambda }}_{k}&\equiv \operatorname {E} [\ln |\mathbf {\Lambda } _{k}|]=\sum _{i=1}^{D}\psi \left({\frac {\nu _{k}+1-i}{2}}\right)+D\ln 2+\ln |\mathbf {W} _{k}|\\\ln {\widetilde {\pi }}_{k}&\equiv \operatorname {E} \left[\ln |\pi _{k}|\right]=\psi (\alpha _{k})-\psi \left(\sum _{i=1}^{K}\alpha _{i}\right)\end{aligned}}}" loading="lazy"></dd></dl>

These results lead to

<msub>

</mrow>

</msub>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

<mrow>

<mrow>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

<mfrac>

<msub>

</mrow>

</msub>

</mfrac>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle r_{nk}\propto {\widetilde {\pi }}_{k}{\widetilde {\Lambda }}_{k}^{1/2}\exp \left\{-{\frac {D}{2\beta _{k}}}-{\frac {\nu _{k}}{2}}(\mathbf {x} _{n}-\mathbf {m} _{k})^{\rm {T}}\mathbf {W} _{k}(\mathbf {x} _{n}-\mathbf {m} _{k})\right\}}</annotation>

</semantics>

</math><img src="./668102a542c2cb9500edffbd07748ff3526257a0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.505ex; width:58.963ex; height:6.176ex;" alt="{\displaystyle r_{nk}\propto {\widetilde {\pi }}_{k}{\widetilde {\Lambda }}_{k}^{1/2}\exp \left\{-{\frac {D}{2\beta _{k}}}-{\frac {\nu _{k}}{2}}(\mathbf {x} _{n}-\mathbf {m} _{k})^{\rm {T}}\mathbf {W} _{k}(\mathbf {x} _{n}-\mathbf {m} _{k})\right\}}" loading="lazy"></dd></dl>

These can be converted from proportional to absolute values by normalizing over <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle k}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle k}</annotation>

</semantics>

Note that:

<ol><li>The update equations for the parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \beta _{k}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \beta _{k}}</annotation>

</semantics>

</math><img src="./026079ab88c28912592bb6e6d2a096f8253a5fba.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.404ex; height:2.509ex;" alt="{\displaystyle \beta _{k}}" loading="lazy">, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {m} _{k}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {m} _{k}}</annotation>

</semantics>

</math><img src="./68d6714a4bc322a9a143d181ba40b467121e27fb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.315ex; height:2.009ex;" alt="{\displaystyle \mathbf {m} _{k}}" loading="lazy">, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {W} _{k}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {W} _{k}}</annotation>

</semantics>

</math><img src="./9102b8a83e111e7056fab952db399d4eab2eb709.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.851ex; height:2.509ex;" alt="{\displaystyle \mathbf {W} _{k}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \nu _{k}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \nu _{k}}</annotation>

</semantics>

</math><img src="./5ff7db3e5f1a1c6ee5a4529f59981e6d4dbf80c6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.237ex; height:2.009ex;" alt="{\displaystyle \nu _{k}}" loading="lazy"> of the variables <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {\mu } _{k}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {\mu } _{k}}</annotation>

</semantics>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {\Lambda } _{k}}</annotation>

</semantics>

</math><img src="./969af1cce7839ff781158a8e64e5643ca85e4f27.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.962ex; height:2.509ex;" alt="{\displaystyle \mathbf {\Lambda } _{k}}" loading="lazy"> depend on the statistics <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle N_{k}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle N_{k}}</annotation>

</semantics>

</math><img src="./7698f968a98115830bcc378e0f849e0375c858c9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.955ex; height:2.509ex;" alt="{\displaystyle N_{k}}" loading="lazy">, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\bar {\mathbf {x} }}_{k}}">

<msub>

<mover>

</mrow>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\bar {\mathbf {x} }}_{k}}</annotation>

</semantics>

</math><img src="./f2d4db4d364c44b97a4a3103f7f6cb58c69951e5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.5ex; height:2.343ex;" alt="{\displaystyle {\bar {\mathbf {x} }}_{k}}" loading="lazy">, and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {S} _{k}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {S} _{k}}</annotation>

</semantics>

</math><img src="./09491d73286bdeeeea46ccc7c69882a9c846a964.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.574ex; height:2.509ex;" alt="{\displaystyle \mathbf {S} _{k}}" loading="lazy">, and these statistics in turn depend on <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle r_{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle r_{nk}}</annotation>

</semantics>

<li>The update equations for the parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \alpha _{1\dots K}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \alpha _{1\dots K}}</annotation>

</semantics>

</math><img src="./aee3ad43735f45d3e35f641e91646c279923584a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:5.928ex; height:2.009ex;" alt="{\displaystyle \alpha _{1\dots K}}" loading="lazy"> of the variable <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {\pi } }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {\pi } }</annotation>

</semantics>

</math><img src="./07b0cee10b7be89207fd2de25f81a23f11018ff8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.332ex; height:1.676ex;" alt="{\displaystyle \mathbf {\pi } }" loading="lazy"> depend on the statistic <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle N_{k}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle N_{k}}</annotation>

</semantics>

</math><img src="./7698f968a98115830bcc378e0f849e0375c858c9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.955ex; height:2.509ex;" alt="{\displaystyle N_{k}}" loading="lazy">, which depends in turn on <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle r_{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle r_{nk}}</annotation>

</semantics>

<li>The update equation for <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle r_{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle r_{nk}}</annotation>

</semantics>

</math><img src="./d2f640f5e142f2e7b490500433a81e863a04413a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.124ex; height:2.009ex;" alt="{\displaystyle r_{nk}}" loading="lazy"> has a direct circular dependence on <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \beta _{k}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \beta _{k}}</annotation>

</semantics>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {m} _{k}}</annotation>

</semantics>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {W} _{k}}</annotation>

</semantics>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \nu _{k}}</annotation>

</semantics>

</math><img src="./5ff7db3e5f1a1c6ee5a4529f59981e6d4dbf80c6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.237ex; height:2.009ex;" alt="{\displaystyle \nu _{k}}" loading="lazy"> as well as an indirect circular dependence on <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {W} _{k}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {W} _{k}}</annotation>

</semantics>

</math><img src="./9102b8a83e111e7056fab952db399d4eab2eb709.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.851ex; height:2.509ex;" alt="{\displaystyle \mathbf {W} _{k}}" loading="lazy">, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \nu _{k}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \nu _{k}}</annotation>

</semantics>

</math><img src="./5ff7db3e5f1a1c6ee5a4529f59981e6d4dbf80c6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.237ex; height:2.009ex;" alt="{\displaystyle \nu _{k}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \alpha _{1\dots K}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \alpha _{1\dots K}}</annotation>

</semantics>

</math><img src="./aee3ad43735f45d3e35f641e91646c279923584a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:5.928ex; height:2.009ex;" alt="{\displaystyle \alpha _{1\dots K}}" loading="lazy"> through <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\widetilde {\pi }}_{k}}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\widetilde {\pi }}_{k}}</annotation>

</semantics>

</math><img src="./216dab116594428a3dd3fd9f33822f8698e3d99b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.421ex; height:2.343ex;" alt="{\displaystyle {\widetilde {\pi }}_{k}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\widetilde {\Lambda }}_{k}}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\widetilde {\Lambda }}_{k}}</annotation>

</semantics>

</math><img src="./adb89aa3345402b19d760fc1227ef5537fa4a442.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.702ex; height:3.176ex;" alt="{\displaystyle {\widetilde {\Lambda }}_{k}}" loading="lazy">.</li></ol>

This suggests an iterative procedure that alternates between two steps:

<ol><li>An E-step that computes the value of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle r_{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle r_{nk}}</annotation>

</semantics>

<li>An M-step that uses the new value of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle r_{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle r_{nk}}</annotation>

</semantics>

Note that these steps correspond closely with the standard EM algorithm to derive a <a href="Maximum_likelihood" class="mw-redirect" title="Maximum likelihood">maximum likelihood</a> or <a href="Maximum_a_posteriori" class="mw-redirect" title="Maximum a posteriori">maximum a posteriori</a> (MAP) solution for the parameters of a <a href="Gaussian_mixture_model" class="mw-redirect" title="Gaussian mixture model">Gaussian mixture model</a>. The responsibilities <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle r_{nk}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle r_{nk}}</annotation>

</semantics>

</math><img src="./d2f640f5e142f2e7b490500433a81e863a04413a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.124ex; height:2.009ex;" alt="{\displaystyle r_{nk}}" loading="lazy"> in the E step correspond closely to the <a href="Posterior_probability" title="Posterior probability">posterior probabilities</a> of the latent variables given the data, i.e. <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p(\mathbf {Z} \mid \mathbf {X} )}">

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p(\mathbf {Z} \mid \mathbf {X} )}</annotation>

</semantics>

</math><img src="./cd89b09f65d1e8a693c636869893e12610a66b8f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.089ex; width:8.659ex; height:2.843ex;" alt="{\displaystyle p(\mathbf {Z} \mid \mathbf {X} )}" loading="lazy">; the computation of the statistics <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle N_{k}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle N_{k}}</annotation>

</semantics>

<msub>

<mover>

</mrow>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\bar {\mathbf {x} }}_{k}}</annotation>

</semantics>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {S} _{k}}</annotation>

</semantics>

</math><img src="./09491d73286bdeeeea46ccc7c69882a9c846a964.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.574ex; height:2.509ex;" alt="{\displaystyle \mathbf {S} _{k}}" loading="lazy"> corresponds closely to the computation of corresponding "soft-count" statistics over the data; and the use of those statistics to compute new values of the parameters corresponds closely to the use of soft counts to compute new parameter values in normal EM over a Gaussian mixture model.

<div class="mw-heading mw-heading2"><h2 id="Exponential-family_distributions">Exponential-family distributions</h2></div>

Note that in the previous example, once the distribution over unobserved variables was assumed to factorize into distributions over the "parameters" and distributions over the "latent data", the derived "best" distribution for each variable was in the same family as the corresponding prior distribution over the variable. This is a general result that holds true for all prior distributions derived from the <a href="Exponential_family" title="Exponential family">exponential family</a>.

<ul><li><a href="Variational_message_passing" title="Variational message passing">Variational message passing</a>: a modular algorithm for variational Bayesian inference.</li>

<li><a href="Variational_autoencoder" title="Variational autoencoder">Variational autoencoder</a>: an artificial neural network belonging to the families of probabilistic graphical models and Variational Bayesian methods.</li>

<li><a href="Expectation%E2%80%93maximization_algorithm" title="Expectation–maximization algorithm">Expectation–maximization algorithm</a>: a related approach which corresponds to a special case of variational Bayesian inference.</li>

<li><a href="Generalized_filtering" title="Generalized filtering">Generalized filtering</a>: a variational filtering scheme for nonlinear state space models.</li>

<li><a href="Calculus_of_variations" title="Calculus of variations">Calculus of variations</a>: the field of mathematical analysis that deals with maximizing or minimizing functionals.</li>

<li>Maximum entropy discrimination: This is a variational inference framework that allows for introducing and accounting for additional large-margin constraints<a href="#cite_note-7">[7]</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFTran2018" class="citation arxiv cs1">Tran, Viet Hung (2018). "Copula Variational Bayes inference via information geometry". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1803.10998">1803.10998</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.IT">cs.IT</a>].</cite>

</li>

<li id="cite_note-Martin2014-2">^ <a href="#cite_ref-Martin2014_2-0">a</a> <a href="#cite_ref-Martin2014_2-1">b</a> <cite id="CITEREFAdamčík2014" class="citation journal cs1">Adamčík, Martin (2014). <a rel="nofollow" class="external text" href="https://doi.org/10.3390%2Fe16126338">"The Information Geometry of Bregman Divergences and Some Applications in Multi-Expert Reasoning"</a>. Entropy. 16 (12): 6338–6381. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2014Entrp..16.6338A">2014Entrp..16.6338A</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.3390%2Fe16126338">10.3390/e16126338</a>.</cite>

</li>

<li id="cite_note-Nguyen-3">^ <a href="#cite_ref-Nguyen_3-0">a</a> <a href="#cite_ref-Nguyen_3-1">b</a> <a href="#cite_ref-Nguyen_3-2">c</a> <cite id="CITEREFNguyen2023" class="citation web cs1">Nguyen, Duy (15 August 2023). <a rel="nofollow" class="external text" href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4541076">"AN IN DEPTH INTRODUCTION TO VARIATIONAL BAYES NOTE"</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.2139%2Fssrn.4541076">10.2139/ssrn.4541076</a>. <a href="SSRN_(identifier)" class="mw-redirect" title="SSRN (identifier)">SSRN</a> <a rel="nofollow" class="external text" href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4541076">4541076</a>. Retrieved 15 August 2023.</cite>

</li>

<li id="cite_note-Yoon2021-4">^ <a href="#cite_ref-Yoon2021_4-0">a</a> <a href="#cite_ref-Yoon2021_4-1">b</a> <a href="#cite_ref-Yoon2021_4-2">c</a> <cite id="CITEREFLee2021" class="citation journal cs1">Lee, Se Yoon (2021). "Gibbs sampler and coordinate ascent variational inference: A set-theoretical review". Communications in Statistics - Theory and Methods. 51 (6): 1–21. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2008.01006">2008.01006</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1080%2F03610926.2021.1921214">10.1080/03610926.2021.1921214</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:220935477">220935477</a>.</cite>

</li>

<li id="cite_note-5"><a href="#cite_ref-5">^</a> <cite id="CITEREFBoydVandenberghe2004" class="citation book cs1">Boyd, Stephen P.; Vandenberghe, Lieven (2004). <a rel="nofollow" class="external text" href="https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf">Convex Optimization</a> (PDF). Cambridge University Press. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-521-83378-3</bdi>. Retrieved October 15, 2011.</cite>

</li>

<li id="cite_note-bishop-6"><a href="#cite_ref-bishop_6-0">^</a> <cite id="CITEREFBishop2006" class="citation book cs1"><a href="Christopher_M._Bishop" class="mw-redirect" title="Christopher M. Bishop">Bishop, Christopher M.</a> (2006). "Chapter 10". Pattern Recognition and Machine Learning. Springer. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-387-31073-2</bdi>.</cite>

</li>

<li id="cite_note-7"><a href="#cite_ref-7">^</a> Sotirios P. Chatzis, “<a rel="nofollow" class="external text" href="http://proceedings.mlr.press/v28/chatzis13.pdf">Infinite Markov-Switching Maximum Entropy Discrimination Machines</a>,” Proc. 30th International Conference on Machine Learning (ICML). Journal of Machine Learning Research: Workshop and Conference Proceedings, vol. 28, no. 3, pp. 729–737, June 2013.

</li>

</ol></div></div>

<div class="mw-heading mw-heading2"><h2 id="External_links">External links</h2></div>

<ul><li><a rel="nofollow" class="external text" href="https://www.inference.phy.cam.ac.uk/mackay/itila/">The on-line textbook: Information Theory, Inference, and Learning Algorithms</a> <a rel="nofollow" class="external text" href="https://web.archive.org/web/20170512025952/https://www.inference.phy.cam.ac.uk/mackay/itila/">Archived</a> 2017-05-12 at the <a href="Wayback_Machine" title="Wayback Machine">Wayback Machine</a>, by <a href="David_J.C._MacKay" class="mw-redirect" title="David J.C. MacKay">David J.C. MacKay</a> provides an introduction to variational methods (p. 422).</li>

<li><a rel="nofollow" class="external text" href="https://www.robots.ox.ac.uk/~sjrob/Pubs/fox_vbtut.pdf">A Tutorial on Variational Bayes</a>. Fox, C. and Roberts, S. 2012. Artificial Intelligence Review, <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2Fs10462-011-9236-8">10.1007/s10462-011-9236-8</a>.</li>

<li><a rel="nofollow" class="external text" href="https://www.gatsby.ucl.ac.uk/vbayes/">Variational-Bayes Repository</a> A repository of research papers, software, and links related to the use of variational methods for approximate Bayesian learning up to 2003.</li>

<li><a rel="nofollow" class="external text" href="https://www.cse.buffalo.edu/faculty/mbeal/thesis/index.html">Variational Algorithms for Approximate Bayesian Inference</a>, by M. J. Beal includes comparisons of EM to Variational Bayesian EM and derivations of several models including Variational Bayesian HMMs.</li>

<li><a rel="nofollow" class="external text" href="https://www.cs.jhu.edu/~jason/tutorials/variational.html">High-Level Explanation of Variational Inference</a> by Jason Eisner may be worth reading before a more mathematically detailed treatment.</li>

<li><a rel="nofollow" class="external text" href="https://arxiv.org/abs/1803.10998">Copula Variational Bayes inference via information geometry (pdf)</a> by Tran, V.H. 2018. This paper is primarily written for students. Via <a href="Bregman_divergence" title="Bregman divergence">Bregman divergence</a>, the paper shows that Variational Bayes is simply a generalized Pythagorean projection of true model onto an arbitrarily correlated (copula) distributional space, of which the independent space is merely a special case.</li>

<li><a rel="nofollow" class="external text" href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4541076">An in depth introduction to Variational Bayes note</a>. Nguyen, D. 2023</li></ul></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-07-25" href="https://en.wikipedia.org/wiki/?title=Variational_Bayesian_methods&oldid=1302454367">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>